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Abstract 


In Fall 2010, the Nlswonger Foundation received a five-year validation grant from the 
Investing in Innovation Fund (i3) to create the Northeast Tennessee College and 
Career Ready Consortium of 29 high schools and five colleges. This report evaluates 
the Consortium’s impact on student outcomes during each of the four years of 
program implementation. The findings from the confirmatory impact analyses 
indicate that students in Consortium schools had higher ACT scores, were more 
likely to participate in Advanced Placement (AP) courses, score a 3 or higher on an AP 
exam, enroll in college, and persist in college than students in matched comparison 
schools. Also, about half of all program components scored 2.0 or higher on a 3- 
point scale, indicating moderate fidelity of implementation. This report contains the 
results submitted to the National Evaluation of i3 (NEi3), which determines the 
overall impact of the federal investment in the i3 program. 
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Executive Summary 


The U.S. Department of Education awarded the Niswonger Foundation a five-year 
Investing in I nn ovation Fund (i3) grant in Fall 2010 to create the Northeast Tennessee 
College and Career Ready Consortium, a network of 29 high schools and five colleges 
across 15 counties. The purpose of the Consortium is to ensure that high school 
students in Northeast Tennessee are provided the opportunity to graduate from high 
school “college or career ready,” and to improve the likelihood that students will be 
successful in college. 

The intervention consists of six components: (1) providing management and 
communication, (2) promoting a college-going culture, (3) increasing quality of 
instruction, (4) increasing access to academically rigorous courses through distance 
and online technology, (5) expanding opportunities for college-level courses, and (6) 
providing resources and services to expand and sustain program capacity. The 
intervention is implemented by Niswonger Foundation leadership, a College and 
Career Ready Counselors Team, a Learning Resources Team, a course review team, 
and a group of online course liaisons. 

These components are designed to achieve specific outcomes: increase students’ 
college and career readiness, enrollment in Advanced Placement (AP) courses, 
performance on AP exams, enrollment in college, and persistence in college. 

Analysis 

This report provides an independent, external evaluation of the impact of the 
intervention on these outcomes through school year (SY) 2014/15, or after up to four 
full years of exposure to the implementation. It also examines the extent to which 
the components of the Consortium were implemented as originally intended in the 
grant application. 

Students in the Consortium schools were compared with students in a group of 
matched comparison schools in school systems in other regions of Tennessee. These 
comparison schools were similar to the Consortium schools in measurable ways 
prior to the start of the grant, but they are not part of the Consortium and do not 
have access to its programs or services. This matched comparison technique 
provides a way to benchmark the progress of students in the Consortium over time. 
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Findings 

Using student-level data, we find statistically significant, positive impacts of the 
Consortium on student composite ACT scores at the end of grade 12 (after three 
years of exposure to the intervention), AP participation by the end of grade 12 (after 
two, three, and four years of exposure to the intervention), AP exam performance 
(after three and four years of exposure to the intervention), and college persistence 
(after two and three years of exposure to the intervention). Consortium students also 
were more likely than comparison students to enroll in college by the fall semester 
following the end of grade 12 (after one year and three years of exposure to the 
intervention), although in year 1 this is likely attributable to differences in the 
samples due to missing pretest data. 

Summary ofthe impactsofthe Niswonger Foundation Bgranton student outcomes. 


Regression Adjusted Mean or 
Predicted Probability 

Yea rs of Consortium Comparison 


Outcome 

Cohort 

exposure 

group 

group 

Difference 

Mean composite ACT 

Grade 11 

2 years 

19.5 

19.5 

0.0 

scoresatthe end of 
grade 12 (Spring 2014) 

Grade 10 

3 years 

19.9 

19.6 

0.3* 

Grade 9 

4 years 

19.7 

19.7 

0.0 

Probability of 

Grade 11 

2 years 

25.6% 

22.8% 

2.8%* 

enrollment in an AP 
course by the end of 
grade 12 (SYs 2012/ 13, 
2013/14, and 2014/15) 

Grade 10 

3 years 

27.8% 

21.5% 

6.3%* 

Grade 9 

4 years 

26.4% 

21.9% 

4.5%* 

Probability of earning a 

Grade 11 

2 years 

9.6% 

8.7% 

0.9% 

score of 3orhigheron 

Grade 10 

3 years 

10.3% 

8.8% 

1.5%* 

an APexam by the 
end of grade 12 (SYs 
2013/ 14 and 2014/15) 

Grade 9 

4 years 

11.1% 

8.7% 

2.4%* 

Probability of 

Grade 12 

1 year 

60.1% 

55.5% 

4.5%* 

enrollment in college 
by the tall semester 
afterthe end of grade 

12 (Fall 2012, Fall 2014) 

Grade 11 

2 years 

58.6% 

56.3% 

2.3% 

Grade 10 

3 years 

57.7% 

54.3% 

3.4%* 

Grade 9 

4 years 

64.6% 

60.6% 

4.0% 

Probability of 

Grade 11 

2 years 

42.5% 

39.5% 

3.1%* 

persistence in college 

Grade 10 

3 years 

40.6% 

37.9% 

2.7%* 


for two c onsec utive 
foil semesters afterthe 
end of grade 12 (Fall 
2014, Fall 2015) 

* Coefficient (of estimated intention-to -treat impact) is statistic a lly significant at the 95 
percent confidence level using a two-tailed test. 
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This means that, on average, students in the Consortium tend to have higher ACT 
scores, are more likely to enroll in an AP course during high school, are more likely 
to earn a score of 3 or higher on an AP exam, are more likely to enroll in college, and 
more likely to persist in college relative to similar students in the comparison group. 

The results from the implementation evaluation indicate that many of the 
Consortium’s components either were implemented with fidelity or were rated very 
close to the threshold of meeting moderate fidelity. The components of the 
intervention that consistently received high implementation ratings throughout the 
portion of the grant period examined were in the areas of Consortium leadership, 
resources and services to provide program infrastructure, and collaboration with 
district partners. 

Summary 

This evaluation provides evidence of the impact of the Niswonger Foundation i3 
grant that may be useful for applying for future grants to scale up or expand upon 
the Consortium’s current activities. 

These results have been submitted to the National Evaluation of i3 (NEi3) to be 
included in a U.S. Department of Education report to Congress on the overall impact 
of the federal investment in the i3 program. The NEi3 report will examine the effects 
of interventions funded by i3 on student outcomes, the strength of the evidence 
generated by the independent i3 evaluations (including CNA’s of the Consortium), 
and the extent to which i3 -funded interventions were implemented with fidelity. 
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Introduction 


In 2010, the U.S. Department of Education sponsored the Investing in Innovation 
Fund (i3) competition to expand a limited number of promising practices with the 
potential to be broadly applied to improve education systems across the country. 
The Niswonger Foundation proposed to create the Northeast Tennessee College and 
Career Ready Consortium, a network of 29 high schools in 15 neighboring districts, 
along with five area colleges. 1 The U.S. Department of Education selected the 
Niswonger Foundation as one of 49 submissions to receive funding from a pool of 
nearly 1,700 submissions. 

Within the i3 program, the Niswonger Foundation i3 grant is classified as a validation 
grant, designed to validate the effectiveness of promising practices for possible 
scale-up elsewhere at a later date. The grant provided the Consortium with $17.8 
million in funding for five years, starting in Fall 2010. The Consortium also secured 
$3.6 million in matching funds from the Niswonger Foundation, the Bill and Melinda 
Gates Foundation, the Rural School and Community Trust, and the J.P. Morgan 
Foundation. 

The Consortium aims to improve high school students’ college and career readiness 
by increasing their access to, participation in, and completion of advanced courses. It 
seeks to achieve these goals by scaling up local promising practices to offer a wide 
array of advanced high school and college credit-bearing courses, particularly in 
mathematics, science, foreign languages, and career and technical education. The 
Consortium uses distance and online learning, as well as college partnerships, to 
increase offerings of Advanced Placement (AP), dual enrollment, and other upper- 
level high school courses. The grant also was used to create a regional coordinating 
body to analyze course supply and demand in the region and determine course 
needs; offer professional development for teachers in an effort to improve the rigor 
of courses; and provide college and career counseling to encourage college access 
and help students with the college application process. 


1 The Consortium added a 30 th school in the 2013/14 school year. This report includes data for 
only the original 29 schools, since we are examining changes in student outcomes and 
implementation of grant activities over the first three years of participation in the grant. 


1 


CNA 

ANALYSIS A SOLUTIONS 


One of the requirements of the i3 program is that each grantee must select an 
external evaluator to conduct an independent analysis of the impact of the 
intervention on educational outcomes and to assess the extent to which the 
intervention was implemented as intended. In the last year of the grant, the 
information must be submitted to the Department of Education’s National Evaluation 
of i3 (NEi3), where it will be included in a report for Congress that aggregates the 
results of all programs funded through i3 grants. These results will be of interest to 
a wide range of stakeholders, including program participants and leadership, federal 
and state policymakers, local administrators and teachers, and the educational 
research community. 

This report provides the findings that were submitted to the NEi3 on the Niswonger 
Foundation i3 grant. 2 It includes results on the impact of the Consortium after up to 
four years of implementation (SY 2011/12 through SY 2014/15) on student 
outcomes of college readiness, AP participation, AP performance, and college 
enrollment. It also provides data on implementation that are used to examine the 
fidelity with which the intervention was carried out as originally intended. 

We begin by providing context for understanding the NEi3 process. Next, we describe 
the components of the Niswonger Foundation i3 grant, as they were originally 
intended to be implemented and how implementation changed over time in the 
intervention. Then we describe the data and methods for the impact and 
implementation evaluations, followed by the findings of each evaluation. We 
conclude with a summary of the results. 


2 This study was conducted under an Institutional Review Board (IRB) exemption from Western 
IRB (WIRB #289900). 
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The National B/aluation of i3 (NE3) 


Puipose of NE3 

The first round of i3 grants was awarded in 2010, consisting of 30 development 
grants (interventions with limited evidence that are implemented on a small scale), 
15 validation grants (interventions with moderate evidence that are implemented on 
a medium scale), and 4 scale-up grants (interventions with strong evidence that are 
being scaled up to other contexts or settings). Each year since, an additional round of 
i3 grants has been awarded, so there currently are five “cohorts” of grantees. 

The Niswonger Foundation i3 grant is classified as a validation grant in the fiscal 
year (FY) 2010 cohort. 

The overarching purpose of the NEi3 is to determine whether the i3 program is 
working (Abt Associates, 2012). It seeks to answer the following questions: 

1. What are the effects of interventions funded by i3 on student outcomes? 

2. What is the strength of the evidence generated by the independent i3 
evaluations? 

3. To what extent were i3 interventions implemented with fidelity? 

Type of information collected for NE3 

The NEi3 is collecting data on each i3 grant, on both the impact of the intervention 
on student outcomes (impact estimates) and the extent to which the intervention was 
implemented as intended (fidelity of implementation). To ensure that the results are 
comparable across a wide range of evaluations, the NEi3 research team developed a 
data collection survey that requires external evaluators such as CNA Education to 
report the results of their evaluations in a consistent manner. NEi3 is using the 
information provided to prepare a project profile for each evaluation, to include a 
description of the characteristics of the intervention implemented by the i3 grantee, 
an assessment of the strength of the evidence provided by the external evaluation, 
and results from analyses to estimate the effect of the intervention. 
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Criteria for assessing the stvength of evidence 

The NEi3 team is assessing the strength of the evidence for each evaluation using 
criteria largely based on the standards defined by the U.S. Department of Education’s 
What Works Clearinghouse (WWC). 

Validation grants such as the Niswonger Foundation i3 grant will receive one of three 
possible ratings for the impact evaluation (Abt Associates, 2012): 

• Meets i3 criteria: This is the highest possible rating, which provides rigorous 
evidence of the impact of the intervention. In order to be eligible for this 
rating, a study must use either a randomized controlled trial design (which 
randomly assigns participants into a treatment group or a control group) or a 
regression discontinuity design (which compares outcomes for students just 
above or below a threshold used to determine participation in the 
intervention). Studies must also meet other criteria defined by the WWC. 

• Meets i3 criteria with reservations: This is the second-highest rating, which 
can be received by studies that use a randomized controlled trial design, a 
regression discontinuity design, or a quasi-experimental design (which uses 
nonexperimental methods to define intervention and comparison groups). 
Studies must also meet other criteria defined by the WWC. 

• Does not meet criteria: This is the lowest rating, which is assigned to grants 
that do not qualify for the ratings of “meets i3 criteria” or “meets i3 criteria 
with reservations.” 

The highest rating that the Niswonger Foundation i3 grant impact evaluation can 
receive is “meets i3 criteria with reservations.” It was not possible to conduct a 
randomized controlled trial for the evaluation, because all of the Consortium schools 
agreed to participate in the intervention prior to the award of the grant and the 
Consortium leadership did not want to withhold services to a subset of the 
Consortium in order to create a randomly assigned comparison group. It also was 
not possible to use a regression discontinuity design, because the Consortium’s 
services are available to all students, not just those who meet certain criteria such as 
a test score requirement. As a result, the CNA team developed a quasi-experimental 
design using a technique known as propensity score matching, which we used to 
identify a comparison group of schools from other regions of Tennessee that are 
similar to the Consortium schools on a comprehensive set of observable 
characteristics. 

There are three additional criteria used to assess the strength of the evidence of the 
external evaluations, as described in Appendix A. CNA designed the impact 
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evaluation for the Niswonger Foundation i3 grant to meet all of the NEi3 criteria 
needed to receive a rating of “meets i3 criteria with reservations.” 

Criteria for assessing the quality of 
implementation findings 

For each i3 grantee, its external evaluator was required to develop a logic model that 
illustrated the key components of the intervention as it was intended to be 
implemented, the mediators (or intermediate outcomes) through which the 
intervention was designed to work, and the program’s expected outcomes. The 
evaluator also was required to develop a fidelity of implementation system that 
defined for each component in the logic model the data sources that could be used 
to measure implementation, a numeric scale for measuring the fidelity of 
implementation, and a criterion for determining whether the component was 
implemented with fidelity. 

These data are helpful for exploring the mechanisms through which the intervention 
achieves its impact (or the lack thereof), supporting or challenging claims of 
causality, and identifying challenges to future implementation and scale-up activities. 

The NEi3 team developed its own criteria for assessing the grantees’ logic models 
and plans for measuring fidelity of implementation, as described in Appendix A. CNA 
designed the implementation evaluation for the Niswonger Foundation i3 grant to 
meet all of the NEi3 criteria for high-quality implementation data. 

limeline for NEB 

The NEi3 began collecting data annually in the spring of 2015. This data collection 
period was timed to allow FY 2010 grants such as the Niswonger Foundation’s to 
provide findings on implementation fidelity and outcomes through school year (SY) 
2013/14, or after three full years of exposure to the implementation. The NEi3’s year 
1 report to Congress is scheduled for release in 2016. The report will consist of an 
individualized project profile with findings for each i3 evaluation, as well as a cross- 
site summary that identifies effective and promising interventions and assesses the 
overall results of the i3 program. 

There will be annual opportunities for a grantee’s external evaluator to report 
findings to the NEi3 even after the end of its grant period. Profiles for each i3 grant 
will be updated in the NEi3’s annual addenda. The first addendum to the report on 
the NEi3 is scheduled for release in the spring of 2017, and a new addendum will be 
released each subsequent spring for as long as the U.S. Department of Education 
continues to fund the i3 program. 
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Description of the Niswonger 
Foundation i3 Grant 


This section summarizes the components of the Niswonger Foundation i3 grant as 
they were intended to be implemented, and discusses the changes that have occurred 
since the grant began in Fall 2010. Prior to the start of SY 2013/14, the Consortium 
experienced significant changes in leadership and organization. As a result, the 
evaluation team, in collaboration with Niswonger Foundation i3 leadership, updated 
its original logic model for the remaining years of the intervention. (Because it is the 
updated logic model that must be reported to NEi3, that is the version described 
below.) 

Description of each component of the 
intervention 

The logic model shown in figure 1 illustrates the key components of the Niswonger 
Foundation i3 grant as it is intended to be implemented in SY 2013/14 and 2014/15, 
the mediators (or intermediate outcomes) through which the intervention is designed 
to work, and the program’s expected outcomes. Differences between the activities 
described in this model and activities from the beginning of the grant (SYs 2011/12 
and 2012/13) are described in the text. 

Below is a list of the key components of that logic model, each of which is described 
in further detail in this section: 

• i3 leadership provides management and communication 

• College and Career Counselors Team promotes a college-going culture 

• Learning Resources Team increases quality of instruction 

• Learning Resources Team increases access to courses through distance and 
online technology 

• Learning Resources Team expands opportunities for college-level courses 

• Resources and services provide infrastructure to expand and sustain 
program capacity 
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Figure 1. Logic model for the Northeast Tennessee College & Career Ready Consortium in SYs 2013/ 14 and 2014/15 




Components 

Consortium Structure, Processes, and Resources 


i3 leadership provides management and 
communication 

• Director convenes the advisory board 

• Attendance is high at board meetings 

• Leadership team meets and shares 
information with staff 

• Leadership team communicates with school 
systems 

• Leadership /Counseling team provides 
outreach to other groups and legislative 
bodies 

• Staff receive professional development (PD) 

• Schools collaborate with each other 

• Conduct course review 

College and Career Counselors Team 
promotes a college-going culture 

• Spend at least 1 day per week at each school 
•Assist students (individually or in groups) 
with developing educational, career, and life 
planning 

•Conduct on site or virtual college visits during 
the school day 

• Conduct on site or virtual college visits 
outside the school calendar 

• Collaborate with educators, community 
members, and parents to assist students with 
educational, career, and life planning 

• Collaborate with other counselors and 
educators in the school-wide integration of 
the guidance services 

Learning Resources Team increases 
quality of instruction 

• Coordinate collaborative professional 
development events and/or resources 

• Coordinate online collaborative professional 
communities 

• Coordinate training for online teachers 

• Assist schools with STEM curriculum and 
instruction 


Learning Resources Team increases access 
to courses through distance and online 
technology 

• Recruit/retain and supervise online teachers 

• Coordinate online course development 

• Coordinate online course evaluation 

• Assist/train school personnel in using 
technology 

• Expand online enrollments 

• Expand distance learning enrollments 

• Ensure schools (students) use ACT Prep 
online software 

Learning Resources Team expands 
opportunities for college level courses 

• Promote or market AP courses 

• Assist in the implementation of AP programs 

• Expand AP enrollments 

• Coordinate dual enrollment with post 
secondary institutions 

• Expand dual enrollment enrollments 

Resources and services provide 
infrastructure to expand and sustain 
program capacity 

• Purchase, maintain, and monitor distance 
learning/conferencing software in schools 

• Maintain openSIS (student information 
system) for online course tracking 

• Maintain learning center website 

• Create/ update course directory as needed 


Mediators 


Changes in high school 
instructional environment 


Wider array of courses available 
More college awareness/ 
readiness opportunities 
More students earn college 
credits prior to high school 
graduation 

More teachers certified to teach 
AP 


Changes in school affective/ 
motivational environment 


Higher expectations for student 
behavior (enroll in rigorous 
courses, apply for college and 
financial aid) 

Greater student motivation to 
successfully complete courses 
and obtain college credit hours 
in high school 


Changes in the 
classroom 


Increase in the use of 
technology for online and 
distance learning 
Increase course quality and 
student engagement 


Changes in the relationship 
between community & school 


• Stronger relationships with 
colleges in the consortium 

• Increase in parental supportfor 
children to attend college 


Outcomes 


Increase college 
readiness ( composite 
scores on the ACT) 

Increase Advanced 

Placement (AP) 
participation (enrollment 
in any AP course) 

I ncrease AP performance 
(score of 3 or higher on 
an AP exam) 

Increase college 
enrollment 

Increase college 
persistence 


CNA 
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i3 leadership provides management and 
communication 

The Consortium’s Executive Director monitors and manages the Consortium’s day-to- 
day operations. The Executive Director is an educator with more than 25 years of 
experience as an assistant principal of curriculum and instruction and a school 
counselor. The Executive Director is supported by the Consortium’s Leadership 
Team, composed of the Niswonger Foundation’s Executive Vice President and the 
Consortium’s Director of Learning Resources, Director of Technology, and 
Compliance Officer. 

The Leadership Team regularly communicates with Consortium staff through 
meetings and with Consortium school systems through correspondence and in- 
person visits. These channels of communication are intended to provide a rich and 
continuous flow of information to the Executive Director and Leadership Team about 
the progress of Consortium initiatives in each of the districts and schools. The 
Leadership Team also conducts outreach to other organizations and policymaking 
entities regarding the Consortium’s activities. 

The Executive Director co-chairs the Niswonger Foundation i3 grant Advisory Board 
with the Foundation’s Executive Vice President. The Executive Director reports 
quarterly to the Advisory Board on the activities of the Consortium and seeks their 
help and guidance to ensure continuous progress toward Consortium goals. The 
Consortium’s Advisory Board is composed of representatives from member schools 
and districts, partner colleges, the Tennessee Department of Education, and 
supporting privately funded organizations. The Board hears reports from the co- 
chairs as well as from the Leadership Team, Learning Resources Team, Director of 
College and Career Counseling, and the external evaluation team. These meetings 
allow Board members to have evidence-based discussions, make recommendations, 
and offer assistance based on the progress made and barriers inhibiting further gains 
toward the Consortium’s goals. High attendance at the Board’s meetings is critical to 
develop buy-in on the Consortium’s activities from all constituents and ensure that 
everyone has a chance to provide feedback. 

In addition, the Executive Director and Leadership Team ensure Consortium staff 
receives professional development each year in areas of interest. The Leadership 
Team also works to have each school collaborate with others in the Consortium, 
through providing an online teacher, sending or receiving a distance learning course, 
or having counselors and teachers collaborate in an online forum. The Leadership 
Team also conducts regular course review meetings at which school and district 
representatives review current courses offered through the Consortium, as well as 
discuss which new courses should be added each year and the preferred delivery 
method for them. 
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College and CareerCounselorsTeam promotesa 
college-going culture 

The Niswonger Foundation i3 grant supports a Director of College and Career 
Counseling and eight College and Career Counselors (CACCs). Each CACC is assigned 
a set of Consortium schools and spends at least one day a week in each of them. 
CACCs work closely with school counselors to support the school-wide integration of 
college and career guidance services. 

The CACCs work directly with students to inform and advise them about high school, 
college, and career topics both individually and in groups. Topics can range from 
ACT™ test preparation and online course taking to completing college applications 
or the Free Application for Federal Student Aid (FAFSA). Many of the topics covered 
also are developed into informational materials intended for students, parents, and 
the community. These materials are shared at outreach events designed to assist 
students with educational, career, and life planning. 

CACCs also set up onsite and virtual college visits for Consortium students both 
during the regular school day and at other times to ensure that the visits are widely 
accessible to students and their parents. 

Learning Resources Team increases quality of 
instruction 

The Learning Resources Team is composed of Consortium staff. Team members have 
individual responsibilities to specific goals of the grant. There is a Director of 
Learning Resources who oversees the team and is responsible for online learning. 
There is also a Director of Professional Learning in charge of providing professional 
development opportunities for teachers and administrators in the Consortium as 
well as staff from the Niswonger Foundation supporting the grant. In addition, there 
are Learning Resources Coordinators for each of the following areas: AP; dual 
enrollment; distance learning; career and technical education (CTE); and science, 
technology, engineering, and mathematics (STEM). These Coordinators work closely 
with Consortium schools to develop, enhance, and expand these various programs 
within each school. 

One major strategy of the i3 grant to improve college and career outcomes for 
Consortium students is to increase the quality of instruction in these schools. The 
Learning Resources Team is responsible for ensuring teachers in Consortium schools 
have collaborative professional learning opportunities offered through the grant. 
Most recently, these opportunities came in the form of teacher academies held in the 
summer for various aspects of the grant or in specific subject areas, as well as 
attendance by AP teachers at College Board trainings in their subject area. The 
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academies provided teachers with information, resources, and opportunities for 
discussion on particular issues such as teaching in a distance learning or online 
setting and teaching an AP course. In addition, opportunities have been provided to 
teachers and administrators to attend meetings of various professional associations 
at the state and national levels. 

Another strategy identified by the Learning Resources Team to improve instructional 
quality is the development of an online collaborative professional community for 
principals, teachers, and counselors. These online communities are designed to give 
educators the opportunity to discuss successes and challenges associated with 
various aspects of their instruction. 

The Learning Resources Team also assists with other tasks designed to improve the 
quality of instruction such as coordinating training for online teachers. This ensures 
that teachers who have never taught online classes before have the training needed 
to properly implement them. In addition, the STEM Coordinator works with schools 
to implement new mathematics and science curriculum. This includes activities such 
as observing classroom instruction and providing feedback, or working with teachers 
to identify curricular resources. 

Learning Resources Team increases access to courses 
through distance and online technology 

Distance and online learning courses are two key strategies used by the Niswonger 
Foundation i3 grant to achieve its goals for expanding students’ access to advanced 
courses in high school. Both delivery methods provide students in remote locations 
with more course options during the school year than they would otherwise have 
available to them through traditional face-to-face courses in their schools. 

The Learning Resources Team handles all aspects of online course development in 
order to expand course enrollments through online technology. They actively recruit 
and supervise online teachers for these courses. In addition, they develop, maintain, 
and evaluate the online courses offered through the grant and assist school 
personnel in the use of the required technology. The Learning Resources Team also 
works with school administrators and other staff to identify and facilitate distance 
learning partnership opportunities in order to expand course enrollments through 
distance learning. 

The Niswonger Foundation i3 grant also provides the Consortium schools with 
access to the ACT Online Prep™ program with the goal of increasing students’ 
college readiness and access. Developed by ACT, Inc., the online preparation program 
includes practice and diagnostic ACT tests, as well as comprehensive review 
materials, in each of the ACT assessment’s four required subject tests— English, 
mathematics, reading, and science. 
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Learning Resources Team expands opportunitiesfor 
college-level courses 

Two other key strategies to improve college and career outcomes for Consortium 
students are to develop and expand AP and dual enrollment programs in schools. AP 
and dual enrollment programs offer rigorous college-level courses and provide 
opportunities for students to gain college credit while in high school. The AP 
Coordinator works closely with principals, teachers, and counselors to create and 
expand AP course offerings and enrollments within a school and promote these 
courses to students and parents. The Coordinator emphasizes working with schools 
that have not previously offered AP courses to their students, so that AP courses 
become more widely available across the Consortium. 

Similarly, the Coordinator works with schools to expand dual enrollment 
opportunities for students. For dual enrollment, the Coordinator collaborates with 
program staff at partner colleges to support implementation of courses in schools, 
including coordination of tuition or textbook assistance. 

Resources and services provide infrastructure to 
expand and sustain program capacity 

The grant funding, as well as in-kind services from participating schools and 
districts, allows the Consortium to afford necessary technology (including high-speed 
Internet, servers, computers, and distance learning equipment), online courses or 
their development, delivery of online courses, and teacher training and stipends. The 
funding also is used for other resources for Consortium courses, such as tuition 
stipends and books for dual enrollment students when there is financial need. The 
Consortium also is responsible for maintaining a functioning distance learning 
website, maintaining a student information system that provides up-to-date online 
course offerings, and creating a course directory schools use to coordinate course 
offerings. 

Changes overtime in the intervention 

While the underlying goals of the Niswonger Foundation i3 Consortium have 
remained consistent from the beginning, Consortium leadership and organization 
significantly changed before the start of SY 2013/14. These changes play a 
significant role in setting the direction of the Consortium in the last two years of the 
grant. 
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In Spring 2013, the Executive Director stepped down. The Consortium’s Director of 
College and Career Counselors was appointed in that role, and the Niswonger 
Foundation’s Executive Vice President took on a larger role in overseeing the 
direction of the project. 

In Summer 2013, the Consortium staff was reorganized in order to better meet the 
needs of the project. A challenge previously had been that no one person was 
responsible for the major strategies of the grant, such as development and expansion 
of AP, distance learning, dual enrollment, and online programs in Consortium 
schools. Instead of a Learning Resources Team, there had been independent 
curriculum specialists who focused their efforts on particular subject areas such as 
English, mathematics, or science, and they planned activities and supports around 
these individual subject areas. Each specialist had contributed to the grant’s 
strategies in different ways, and it was difficult to attribute successes and challenges 
to specific activities of the staff. 

As a result of the reorganization, the role of the curriculum specialists was changed 
to Learning Resources Coordinators, with each responsible for one of the major 
strategies of the Consortium. Ideally, this reorganization would ensure that someone 
was actively working toward the specific goals of the Consortium. The career 
counseling staff also was realigned to ensure consistency and continuity of services 
through the remainder of the grant period. 

Despite the leadership and organizational changes, however, many of the underlying 
supports to schools and students, such as professional development opportunities 
for staff, dual enrollment tuition assistance, and the type of college and career 
counseling provided for students, remained largely unchanged. 

Meantime, the Consortium has been able to expand support to districts and schools 
through a mini-grant program. These mini-grants began in Summer 2013 as a way for 
schools to individually work toward Consortium goals. In their proposals to the 
Consortium leadership, schools identified objectives of the support they requested, 
how the support related to Consortium goals, and a timeline, budget, and 
sustainability plan. Between July 1, 2013, and May 1, 2014, the Consortium approved 
62 mini-grant proposals with a total funding amount of $1.3 million. Examples of 
projects funded include technology purchases (e.g., computers, laptop carts, 
software, and calculators), resources for starting up new AP courses, and support for 
summer “boot camps” and afterschool study programs. The mini-grant program was 
not originally a part of the i3 grant proposal or logic model, but Consortium staff 
saw opportunities to support innovation specific to individual schools. 
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Data and Methods 


This section describes the data and methods for both the impact evaluation and the 
implementation evaluation. 

Funding for the Consortium began in Fall 2010, but SY 2010/11 was used primarily 
for planning and building necessary program infrastructure. Thus, SY 2010/11 
serves as the baseline period, because it represents student and school conditions 
prior to the intervention. Program implementation, including teacher training for AP 
instruction and the addition of online courses for Consortium students, began in 
Summer 2011. The first full year of implementation, therefore, is SY 2011/12. This 
means that during the five-year grant period, there are four years of implementation 
from which data can be collected (Fall 2011 through Fall 2014). The Niswonger 
Foundation also received a one-year extension of the evaluation period to collect 
college enrollment and persistence data through Fall 2015. 

Impact eva luation 

The impact evaluation seeks to determine whether the Niswonger Foundation i3 
grant is an effective approach for increasing college readiness, AP participation, AP 
performance, college enrollment, and college persistence. To assess the Consortium’s 
effectiveness, the evaluation team compared outcomes in these domains for students 
in Consortium schools with those of students in similar schools that are not part of 
the Consortium. 

It is important to note that these impacts should not be interpreted as causal, as 
there may be unobserved differences between the Consortium and comparison 
groups that are not accounted for in the analysis and may influence student 
outcomes. 

Research questions for the impact evaluation 

The NEi3 categorizes research questions into two categories: confirmatory and 
exploratory. Confirmatory research questions answer the main policy questions 
using the most rigorous design possible (Burghardt, Deke, Kisker, Puma, and 
Schochet, 2009). Exploratory research questions provide an opportunity to further 
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examine relationships in the data and examine how the impact of the intervention 
differs for various groups of students, teachers, or schools. The evaluation of the 
Niswonger Foundation i3 grant includes both confirmatory and exploratory research 
questions, as described below. 

Confirmatory research questions 

The confirmatory research questions are based on four domains of interest: college 
readiness, AP performance, college enrollment, and college persistence. The 
confirmatory research questions include the full sample of students from the cohort 
with the longest duration of exposure to the intervention at the time of the last 
possible data collection during the grant period. These cohorts were selected for the 
confirmatory analyses because we expect to find the greatest impact of the 
intervention for students with the most exposure to the intervention. There are four 
confirmatory research questions: 

What is the impact of the Niswonger Foundation i3 grant on 

1. College readiness, as measured by composite scores on the ACT by the end of 
grade 12 (after up to three years of exposure to the intervention)? 

2. AP performance, as measured by an earned score of 3 or higher on an AP exam 
by the end of grade 12 (after three years of exposure to the intervention)? 

3. College enrollment, as measured by enrollment in a degree-granting two-year 
or four -year college or at a Tennessee Technology Center by the fall semester 
in the year after high school graduation (after three years of exposure to the 
intervention)? 

4. College persistence, as measured by enrollment in a degree-granting two-year 
or four-year college or at a Tennessee Technology Center in the fall semester 
one year and two years after high school graduation (after two years of 
exposure to the intervention)? 

After the initial grant period ended, the Niswonger Foundation received a one-year 
no-cost extension of the evaluation period to collect an additional year of data on all 
four outcomes. 3 These additional results were submitted to NEi3’s first addenda. 


3 While the additional year of data for each of these outcomes represents the longest duration 
of exposure to the intervention, they are not classified as confirmatory or exploratory because 
they were added after the data collection for the original grant period. 
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Exploratory research questions 

The specific questions to be addressed in the exploratory analysis are the following: 

1. What is the impact of the Niswonger Foundation i3 grant for students with 
fewer years of exposure to the treatment on: 

a. College readiness (after two and three years of exposure to the 
intervention)? 

b. AP performance (after two and three years of exposure to the intervention)? 

c. College enrollment (after one year and two years of exposure to the 
intervention)? 

2. Does the impact of the Niswonger Foundation i3 grant differ for student 
subgroups (free and reduced-price lunch status, CTE concentrator status) and 
school locale (rural/town versus city/suburban)? 

3. What is the impact of the Niswonger Foundation i3 grant on the additional 
domain of AP participation, as measured by enrollment in one or more AP 
courses by the end of grade 12? 

4. What is the impact of the Niswonger Foundation i3 grant on AP exam 
performance for the subgroup of students who enrolled in an AP course? 

5. What is the impact of the Niswonger Foundation i3 grant on ACT scores at the 
end of grade 11 (after three years of exposure to the intervention)? 

The first exploratory research question addresses whether the Niswonger Foundation 
i3 grant had an impact on outcomes for cohorts with fewer years of exposure to the 
intervention. The effectiveness of educational interventions can vary by exposure and 
dosage (Century, Rudnick, & Freeman, 2010). For example, Consortium and school 
personnel will gain more experience over time and may become more effective in 
implementing the intervention’s activities. Conducting subgroup analyses for cohorts 
of students with different years of exposure to the intervention will improve our 
understanding of the conditions under which the intervention may work. This 
information also would be useful if the intervention were scaled up or replicated, so 
that implementers would know when they might expect to begin finding changes in 
student outcomes. 

The second exploratory research question examines whether the impact of the grant 
differs for student and school subgroups. These subgroup analyses are conducted 
only if there is an overall effect of the intervention on student outcomes. Research 
shows that college enrollment and persistence rates tend to be lower for low-income 
students (Bastedo & Jaquette, 2011; Tinto, 2006) and students in rural schools 
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(Doyle, Kleinfeld, & Reyes, 2009; McDonough & McClafferty, 2001); so we may expect 
students in these subgroups to respond differently to the intervention than their 
peers do. In addition, the Consortium’s counselors note that students who complete 
high school CTE concentrations often believe that they do not need postsecondary 
education to find a job in their CTE program area. The counselors have been advising 
these students to at least enroll in a short-term postsecondary certificate program in 
their CTE program area, so outcomes for CTE concentrators may differ in 
Consortium schools and comparison schools. 

The third question assesses whether the Niswonger Foundation i3 grant has an 
impact on an additional outcome for AP participation. This outcome is categorized as 
exploratory because the literature provides mixed evidence for the association 
between college outcomes and participation in AP courses for students not taking an 
AP exam (Adelman, 2006; Dougherty, Mellor, & Jian, 2006; Geiser & Santelices, 2004; 
Hargrove, Godin, & Dodd, 2008; Mattern, Shaw, & Xiong, 2009; Speroni, 2011). 
Evidence is also mixed on whether students benefit from taking an AP course and 
exam even if they do not pass the exam (Dougherty, Mellor, & Jian, 2006; Hargrove, 
Godin, & Dodd, 2008; Jackson, 2010; Mattern, Shaw, & Xiong, 2009). 

The fourth question examines the impact of the Niswonger Foundation i3 grant on 
AP exam performance for the subgroup of students enrolled in an AP course. This 
analysis examines whether students in Consortium schools who enroll in AP courses 
perform better than students in comparison schools in AP courses do. There are 
several reasons why an effect may be anticipated here. Consortium students may be 
better prepared academically because they may have greater access to rigorous 
courses prior to their enrollment in AP courses. The Consortium also provides 
professional development to AP teachers, which may enable them to teach their 
courses more effectively. 

The fifth exploratory research question examines the impact of the Niswonger 
Foundation i3 grant on ACT scores at the end of grade 11 instead of the end of grade 
12. The original analysis plan for the evaluation had grade 11 ACT scores as the 
confirmatory outcome. We obtained SY 2011/12 and 2012/13 ACT scores from the 
Tennessee Department of Education, but a preliminary examination of the data 
showed that 41 percent of students in the sample were missing ACT scores in grade 
11. The state also provided a file with ACT scores for high school graduates, which 
includes ACT scores from tests taken at any time in high school. This file had a much 
lower missing data rate of 20 percent. Since there are fewer problems with missing 
data in the graduates file, we revised our plan, examining the ACT outcome at the 
end of high school (grade 12)— instead of ACT scores in grade 11— for the 
confirmatory analysis. We, however, still include the grade 11 ACT scores as an 
exploratory outcome in accordance with the scientific process requirements outlined 
under “other relevant factors for assessing i3 evaluations” in Appendix A of this 
report describing the NEi3. 
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Souncesof data forthe impact evaluation 

The outcome measures were selected because they are manifestations of the goals of 
the Consortium, as shown in the logic model. Table 1 describes each of the outcome 
variables for the impact evaluation, including the variable type (continuous or 
dichotomous), the data source, and the timing of the variable. 

Table 1. Summary of outcome variables for the impact evaluation, by research 

domain. 



College 
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AP 

Performance 
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Pa rtic ipation Enrollment 
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The Tennessee Department of Education provided data on college readiness, AP 
performance, and AP participation for all students statewide; the Tennessee Higher 
Education Commission (THEC) provided college enrollment records. THEC collects 
college enrollment data from the National Student Clearinghouse, which has records 
from more than 3,600 colleges and universities, accounting for more than 98 percent 
of all students enrolled in public and private U.S. postsecondary institutions. 4 THEC 
also supplements the National Student Clearinghouse data with its own records on 
students who attend a Tennessee College of Applied Technology (TCAT) or any other 
public, in-state postsecondary institutions that do not participate in the National 
Student Clearinghouse. 5 THEC also has data on enrollment at in-state private colleges 
for recipients of Tennessee Hope, a lottery-funded academic scholarship program for 
in-state colleges. THEC also provided data on college persistence during the fall 
semester one year after grade 12 for students enrolled at in-state colleges and TCAT. 
However, data from the National Student Clearinghouse were not available for the 
college persistence outcome, so students are excluded from the analyses if they were 
enrolled in a postsecondary institution from the National Student Clearinghouse 
records during the fall semester immediately following grade 12. 

The cohorts of students eligible for the impact evaluation differ for each research 
question because the amount of follow-up time required to collect data differs by 
outcome, as shown in table 2. The cohorts selected for the confirmatory analysis 
represent students with the longest exposure to the intervention in the time available 
for the evaluation period. 

Students in grade 12 in SY 2010/11 had only one year of exposure to the 
intervention. There may be an impact on college enrollment because these students 
had access to the Consortium’s College and Career Counselors in their senior year. 
These students, however, had low exposure to the intervention, with few 
opportunities to take the new rigorous courses added by the Consortium, so no 
impact is expected on ACT scores, AP outcomes, or college persistence. Thus college 
enrollment is the only outcome analyzed for this cohort. 


4 http://www.studentclearinghouse.org/about/ 

5 Tennessee has a statewide system of 27 Tennessee Colleges of Applied Technology (TCATs) 
that offer one- to two-year technical/occupational postsecondary programs (see 
http://www.completecollege.org/docs/Tennessee%20Technology%20Centers- 

%20A%20Preliminarv%20Case%20Studv(l).pdf) . 
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Table 2. List of cohortsand outcomesto be analyzed foreach yearof the 
intervention. 


Cohort 

2011/12 

Intervention 

Yearl 

2012/13 
Intervention 
Year 2 

2013/14 
Intervention 
Year 3 

2014/15 

Intervention 

Year4 

Fall 2015 
Post 

Intervention 

Grade 9 

Grade 9 

Grade 10 

Grade 11 

ACTscores 

(exploratory) 

Grade 12 

ACTscores 

(exploratory) 

AP exam 3+ 
(confirmatory) 

AP course 
enroll 

(exploratory) 

College 

yearl 

College 

enroll 

Grade 

10 

Grade 10 

Grade 11 

Grade 12 

ACTscores 

(confirmatory) 

AP exam 3+ 
(confirmatory) 

AP course 
enroll 

(exp loratory) 

College yearl 

College enroll 
(confirmatory) 

College 
year 2 

College 

persist 

Grade 

11 

Grade 11 

Grade 12 

ACTscores 

(exploratory) 

AP exam 3+ 
(exploratory) 

AP course 
enroll 

(exp loratory) 

College yearl 

College enroll 
(exploratory) 

College year 2 

College persist 
(confirmatory) 


Grade 

12 

Grade 12 

College yearl 

College enroll 
(exp loratory) 

College year2 

College year3 



Methods for assessing the impact of the intervention 
on student outcomes 

The intervention group consists of 29 high schools in 15 school districts that agreed 
to participate in the Consortium prior to the award of the grant. This is a 
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convenience sample of districts in Northeast Tennessee that had previously worked 
in partnership with the Niswonger Foundation on other projects. 

For the impact evaluation, a school-level propensity score model was used to identify 
a matched comparison school for each of the intervention schools. This is important 
because there may be other changes over time independent of the i3 grant that may 
influence student outcomes, such as activities under Tennessee’s Race to the Top 
award or the adoption of new Tennessee Core Standards. The comparison group 
provides a way to benchmark the progress of the Consortium against other schools 
that are subject to the same changes in state policy over time. The comparison 
schools operate under business-as-usual conditions, without membership in the 
Consortium or access to the Consortium’s activities or resources. 

All regular public high schools in the state of Tennessee were included in a 
propensity score model used to identify the 29 comparison schools. This statistical 
technique requires estimating a logistic regression equation of the likelihood that a 
school is a member of the Consortium, controlling for demographic characteristics of 
the student population, test scores of the student population, school resources, and 
community characteristics. From this equation we calculate a score for each school, 
based on those characteristics, and then match Consortium schools to non- 
Consortium schools that have the closest scores. The impact analysis compares the 
outcomes for students enrolled in Consortium schools with outcomes for students 
enrolled in schools with similar characteristics in the comparison group (see 
Appendix B for additional information on the methods for propensity score 
matching). 

The matched comparison schools identified from the propensity score model had to 
be recruited to participate in the evaluation. The primary responsibility of the 
comparison schools is to participate in a bi-annual survey about course enrollments. 
The comparison schools also were asked to allow the evaluation team to conduct 
classroom observations in the first and last years of the grant. These data provide 
descriptive formative feedback on program implementation and are not included in 
the impact evaluation. Comparison schools do not need to provide outcome data for 
the impact evaluation because data about their students’ outcomes are available 
statewide from administrative records. 

The Consortium staff called the principals at all 29 matched comparison schools to 
request their participation in the study, offering them an annual financial incentive. 
Only one school declined to participate. This school was replaced with the school 
with the next-highest propensity score value. The final comparison group consists of 
29 high schools in 20 districts. 

The analysis sample for the impact evaluation consists of all students enrolled in a 
Consortium school or a matched comparison school at the beginning of the first year 
of program implementation (as of October 1, 2011) who have outcome and pretest 
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data. Regression models are used to estimate the impact of the intervention while 
controlling for students’ pretest scores, school-level average pretest scores, student 
demographic characteristics, and school characteristics (see Appendix B for 
additional information on the model specification). Inclusion of these covariates in 
the models reduces bias in the estimate of the intervention’s effect and improves the 
precision of the estimates. 

If students in Consortium schools show positive and statistically significant 
differences relative to the comparison schools, this will indicate that the 
Consortium’s programs and activities may have had a positive impact on its 
students’ college and career readiness. 

Baseline equivalence of the impact evaluation 
sample 

After estimating the school-level propensity score model, we conducted a preliminary 
analysis to establish whether the intervention and matched comparison groups were 
similar at baseline. We compared the average values for the two groups on a set of 
observed school-level factors representing student and community characteristics, 
test scores of the student population, attendance and graduation rates, and school 
resources. 

After matching, there are no statistically significant differences between the 
Consortium schools and the matched comparison schools on any of the observed 
characteristics (additional details provided in the Appendix B). 

The most important comparison for the impact evaluation is whether student 
outcomes prior to the i3 grant differ between the Consortium schools and the 
matched comparison schools. Student pretest scores are the primary indicator of 
baseline equivalence for each of the outcome measures. 

The specific tests for baseline equivalence differ depending on the grade level of the 
students in the sample prior to the start of the grant, and include both state and 
national assessments. The state assessments are the Tennessee Comprehensive 
Assessment Program (TCAP) exams and end-of-course (EOC) subject exams 
administered at the end of high school courses in the corresponding subject area. 
National assessments used for the pretests are PLAN™ (grade 10) and ACT™ (grade 
11), which are part of the ACT’s Educational Planning and Assessment System. All 
pretests were administered statewide in SY 2010/11, the school year prior to the 
start of full implementation of the i3 grant in Fall 2011. 

Baseline equivalence is established if the mean difference between pretest scores for 
students in the intervention group and the comparison group is less than or equal to 
0.05 standard deviation unit, or if the difference is greater than 0.05 standard 
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deviation unit but less than 0.25 standard deviation unit and the pretest is included 
in the impact model. 6 Table 3 shows that the standardized intervention-comparison 
differences in pretests are less than 0.25 standard deviation unit for all cohorts and 
outcomes. Because there are still small baseline differences for most variables, 
however, we include pretest covariates in all statistical models of the impact of the 
intervention. 

Table 3. Test of baseline equivalence of the intervention and comparison groups 
on pretest measures. 


Cohort 

Pretest 

Outcome 

Intervention 

Comparison 

Difference 

Effect Size 
Difference 

Included 

in 

Impact 

Model? 

Grade 

9 

Grade 8 
1CAP in 
reading & 
math 

ACTscores(gr. 11) 

13.813 

(reading) 

16.309 

(math) 

0.125 

(reading) 

0.147 

(math) 

Yes 



ACTscores (gr. 12) 

9.584 

(reading) 

0.099 

(reading) 

Yes 




12.188 

(math) 

0.125 

(math) 




AP participation 

9.366 

(reading) 

0.082 

(reading) 

Yes 




11.660 

(math) 

0.101 

(math) 




AP performance 

9.366 

(reading) 

0.082 

(reading) 

Yes 




11.660 

(math) 

0.101 

(math) 




College enrollment 

-4.859 

(reading) 

0.076 

(reading) 

Yes 


6 In order to determine baseline equivalence, we estimate a two-level hierarchical model with 
students nested in schools, where the dependent variable is the pretest score and there is one 
independent variable for intervention status (0=matched comparison group, and 1 Consortium 
group). This model has the same structural components as the impact model but does not 
include the other student- and school-level covariates. The coefficient on the intervention 
variable represents the estimated intervention-comparison group difference in pretest measure 
after adjusting for the nested nature of the data. This difference is divided by the standard 
deviation of the outcome for the control group to compute the baseline difference in effect-size 
units. 
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Cohort 

Retest 

Outcome 

Intervention 

Comparison 

Difference 

Effect Size 
Difference 

Included 

in 

Impact 

Model? 




-7.437 

(math) 

0.095 

(math) 


Grade 

Grade 9EOC 

ACTscores (gr. 12) 

2.240 

0.062 

Yes 

10 

test in English 

AP participation 

-0.922 

-0.024 

Yes 



AP performance 

-0.922 

-0.024 

Yes 



AP performance 
(subgroup of students in 
an AP course) 

-3.273 

-0.139 

Yes 



College enrollment 

-0.922 

-0.024 

Yes 



College persistence 

-1.266 

-0.033 

Yes 

Grade 

Grade 10 

ACTscores (gr. 12) 

0.201 

0.055 

Yes 

11 

PLAN 

composite 

score 

AP performance 

0.019 

0.005 

Yes 


AP performance 
(subgroup of students in 
an AP course) 

-0.310 

0.091 

Yes 



AP participation 

0.019 

0.001 

Yes 



College enrollment 

0.019 

0.005 

Yes 



College persistence 

-0.018 

-0.005 

Yes 

Grade 

12 

Grade 11 

ACT 

composite 

score 

College enrollment 

0.186 

0.039 

Yes 


Implementation evaluation 

Methods for assessing fidelity of implementation 

The second part of the evaluation assesses the fidelity of implementation for the 
intervention. The logic model (see figure 1) illustrates the key components of the 
Niswonger Foundation i3 grant as it is intended to be implemented. Fidelity of 
implementation is measured separately for each of these components in each year of 
the intervention. Due to changes over time in the measurement of some of the 
indicators, however, the results are not directly comparable across all years. This 
report includes data from SYs 2011/12 to 2014/15. 

To calculate the numeric fidelity score for a component required looking separately 
at the fidelity of each of several indicators comprising that component. Each 
indicator was rated by the evaluation team on a 3-point scale where l=“low,” 
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2=“moderate,” and 3=“high” implementation. In SYs 2011/12 and 2012/13, each 
indicator also was assigned a weight based on the relative importance of that activity 
in achieving the objective of the larger component. These weights were developed in 
consultation with Consortium leadership and applied as percentages so that the sum 
of all indicators within a component added up to 100 percent. Finally, a weighted 
average of the fidelity scores for all indicators was calculated for each component. 

Components are defined as being implemented “with fidelity” if this weighted 
average is greater than or equal to 2.5 on a 3-point scale. While this benchmark is 
somewhat arbitrary, it was determined prior to the collection of the fidelity data, so 
it was not selected to make the findings appear more favorable. 

With the change in Consortium leadership in Summer 2013 came changes to the 
structure of the components. Given that, in SY 2013/14, each indicator was weighted 
equally in the average fidelity score for each component. Fidelity still was determined 
using the 2.5 benchmark, however. Again, all changes to the scoring scheme were 
made prior to the collection of the fidelity data. 

Sources of data for assessing fidelity of 
implementation 

The fidelity of implementation for the Consortium was measured using multiple 
sources of data, most of which fall into one of five categories. First, all Consortium 
staff members regularly kept activity logs that included the schools they worked 
with, the activities they completed, and the amount of time they spent on each task. 
The activity logs were sent to the evaluation team at least once per semester for 
inclusion in the implementation analysis. Second, the evaluation team kept notes 
from the Consortium Executive Director’s regular meetings with the staff and with 
the evaluation team. The type of information collected during these meetings 
included any hiring and retention changes in Consortium staff and updates from the 
Director reporting on presentations to outside groups. 

Third, each year the Compliance Officer sent the evaluation team copies of the 
annual report submitted to the U.S. Department of Education as part of the i3 grant’s 
federal reporting requirements. These reports contain information such as the 
amount of money spent on various resources and the extent to which districts are 
picking up recurring costs. Fourth, the evaluation team requested copies of 
deliverables produced by Consortium staff, such as the Learning Resources Team’s 
course directory and the College and Career Counselors’ Consortium-wide 
informational materials. 

Fifth, the evaluation team collected its own data through surveys and interviews. 
Each year the evaluation team administered a survey to all Consortium high schools 
with questions about the names of advanced courses offered, course enrollment, and 
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the mode of delivery. In addition, the evaluation team conducted interviews with 
Consortium school and college staff by phone or during site visits about barriers to 
and progress toward program implementation. 
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findings Impact of the Intervention 


This study examines the effects of the Niswonger Foundation i3 grant on student 
outcomes for (1) college readiness, (2) AP participation, (3) AP performance, (4) 
college enrollment, and (5) college persistence. This chapter presents the impact 
findings for each of these outcomes based on multilevel models that account for the 
clustering of students within schools. We also discuss the results of additional 
exploratory analyses conducted to examine the impact of the intervention for 
cohorts of students with fewer years of exposure to the intervention, for student and 
school subgroups, and for a subsample of the full analysis sample. 

For each outcome, we provide an illustrative graphical display of the impact results 
using bar charts. For the continuous outcome variable (composite ACT scores), a bar 
is provided with the unadjusted mean for students in the comparison group. A 
second bar is provided for the Consortium group that adds the coefficient of the 
intervention variable from the multilevel model to the comparison group mean. For 
the remaining dichotomous variables, the value for the comparison group in the bar 
chart is computed by using the estimated equation to calculate each individual 
student’s probability of a successful outcome (enrolling in an AP course, earning a 
score of 3 or higher on an AP exam, or enrolling in college) using his or her 
individual characteristics and setting the value of the intervention variable to 0 for 
everyone in the sample. We then repeated the process, setting the value of the 
intervention variable to 1 to calculate the predicted probability for students in the 
Consortium group. 

The notes at the bottom of each figure provide the p-value, which is used to assess 
the statistical significance of the impact estimate. The p-value represents the 
likelihood of obtaining a finding due to random chance rather than to the 
investigated effect. Using a two-tailed test, we consider an impact to be statistically 
significant if there is less than a 5 percent chance of obtaining a random finding (a p- 
value of less than 0.05). The figures also include an asterisk next to the value for the 
Consortium if the results are statistically significant. 

More detailed results from the regression models can be found in Appendix C, Table 
10. This table shows how the results were presented for the NEi3 report. 
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College readiness 

The composite scores on the ACT at the end of grade 12 are used as the primary 
indicator of college readiness. The confirmatory analysis for college readiness uses 
the sample of students from the grade 10 cohort, who had up to three years of 
exposure to the intervention at the time of the outcome (Spring 2014). This is the 
longest follow-up period possible within the timeframe of the original grant. During 
the extension of the evaluation period, an additional year of data was added for the 
grade 9 cohort, who had a full four years of exposure to the intervention. 

We also conduct an exploratory analysis of the impact of the intervention on college 
readiness (composite ACT scores) for the sample of students from the grade 11 
cohort, who had up to two years of exposure to the intervention at the time of the 
outcome (Spring 2013). This analysis is considered exploratory because the students 
had fewer years of exposure to the intervention. The Niswonger Foundation i3 grant 
did not have a statistically significant impact on college readiness in Spring 2013 for 
students with up to two years of exposure to the intervention (figure 2). The 
regression-adjusted mean is 19.5 for students in both the comparison and 
Consortium groups. There was also no statistically significant impact on college 
readiness in Spring 2015 for students with four years of exposure to the 
intervention. The regression-adjusted mean is 19.7 for both groups. 

For students with up to three years of exposure to the intervention, however, the 
intervention is associated with a statistically significant impact on college readiness 
in Spring 2014. For this cohort, the regression-adjusted mean is 19.6 for the 
Consortium group and 19.9 for the comparison group, a difference of approximately 
0.3. The pooled standard deviation for the sample is 0.25, which means that the 
effect size is 0.05 standard deviation unit. 7 

Although the effect size for the Niswonger Foundation i3 grant is small in 
magnitude, it is statistically significant, with a p-value of 0.004. This means that 
students in the Consortium group achieved higher composite scores on the ACT at 
the end of grade 12 than did students in the comparison group, after up to three 
years of exposure to the intervention. 


7 In comparison, a study that summarized 468 achievement effect sizes from random 
assignment studies of educational interventions found that the mean effect size was 0.27 
standard deviation unit for studies of high schools (Hill, Bloom, Black, & Lipsey, 2008). 
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Figure 2. Regression-adjusted mean composite ACTscoresat the end of grade 12 
forstudents in the comparison and Consortium groups, Spring 2013, Spring 
2014, and Spring 2015. 



Spring 2013 Spring 2014 Spring 2015 

(2 years exposure) (3 years exposure) (4 years exposure) 


Notes: Resultsare from a two-level hierarchical linear model that accommodatesfor 
clustering of students within schools. The value for the comparison group is the unadjusted 
mean. The value forthe Consortium group addsthe estimated intention-to-treat impact 
from the model to the comparison group mean. The estimated intention-to-treat impact is 
0.019 (p =0.760) in Spring 2013,0.246 (p =0.004) in Spring 2014, and -0.034 (p =0.772) in Spring 
2015. 

Coefficient (of estimated intention-to-treat impact) is statistic a lly significant at the 95 
percent confidence level using a two-tailed test. 


We also examined whether the Impact of the intervention on composite ACT scores 
after up to three years of exposure differed by student subgroups (free and reduced- 
price lunch, CTE concentrator status) and by school subgroup (locale ). 8 There is no 
difference in the impact of the intervention by either of the student subgroups. But 
we did find that the impact of the intervention is greater for students in non-rural 


8 School locale is defined using the urban-centric locale codes of the National Center for 
Education Statistics (see http://nces.ed.gov/ccd/mral_locales.asp ). For this analysis, we 
compare schools located in rural and town locales with schools located in suburb and city 
locales. 
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schools. The regression-adjusted mean composite ACT scores are about half a point 
higher for students in non-rural Consortium schools compared with students in non- 
rural comparison schools. For students in rural schools, the regression-adjusted 
mean composite score is a tenth of a point lower for the Consortium group relative 
to the comparison group. 

An additional exploratory analysis examined the impact of the Niswonger Foundation 
i3 grant on composite ACT scores at the end of grade 11 instead of at the end of 
grade 12. The sample for this analysis is students in the grade 9 cohort with three 
years of exposure to the intervention who took the ACT exam in Spring 2014. The 
results indicate that the Niswonger Foundation i3 grant does not have a statistically 
significant impact on college readiness for this sample. The regression-adjusted 
mean is 19.0 for students in the comparison group and 18.9 for students in the 
Consortium group (figure 3). 

Figure 3. Regression-adjusted mean composite ACTscoresat the end of grade 11 
for students in the comparison and Consortium groups, Spring 2014. 


36.0 

31.0 

26.0 

21.0 

16.0 

11.0 

6.0 

1.0 


■ Comparison Consortium 


19.0 18.9 



Spring 2014 
(3 years exposure) 


Notes: Resultsare from a two-level hierarchical linear model that accommodatesfor 
clustering of students within schools. The value for the comparison group is the unadjusted 
mean. The value forthe Consortium group addsthe estimated intention-to-treat impact 
from the model to the comparison group mean. The estimated intention-to-treat impact is 
-0.086 (p =0.492). 
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Advanced Placement participation 

We took a similar approach to estimate the impact findings for enrollment in one or 
more AP courses by the end of grade 12, an indicator of students’ AP participation. 
This was a dichotomous variable for whether the student enrolled in an AP course at 
any time during the intervention period (l=yes, 0=no). Because this was a 
dichotomous dependent variable, we estimated the model using a logit link function, 
specifically a hierarchical generalized linear model. 

This outcome is categorized as exploratory because the literature provides mixed 
evidence for the association between college outcomes and participation in AP 
courses for students not taking an AP exam. The effects are estimated for students 
who have up to four years of exposure to the intervention. 

The analysis for SY 2012/13 includes the sample of students in the grade 11 cohort 
who had two years of exposure to the intervention; the analysis for SY 2013/14 
includes the sample of students in the grade 10 cohort who had three years of 
exposure to the intervention; and the analysis for SY 2014/15 includes the sample of 
students in the grade 9 cohort who had a full four years of exposure to the 
intervention. 

We find that the Niswonger Foundation i3 grant does have a statistically significant 
impact on participation in AP courses in all three years. In SY 2012/13, the 
comparison group probability of 22.8 percent represents, for students in the sample, 
the probability that a student from a comparison school will enroll in an AP course 
by the end of grade 12 (figure 4). 

The predicted probability for students in the Consortium schools is 25.6 percent, a 
difference of approximately 3 percentage points. In SY 2013/14, the predicted 
probability that a student will enroll in an AP course increases to 27.8 percent, while 
the comparison group remains relatively constant at 21.5 percent. Results are similar 
for 2014/15, with a predicted probability of 26.4 percent for students in the 
Consortium, compared with 21.9 percent in the comparison group. 

The difference between the two groups ranges from 2.8 percent in 2012/13 to 6.3 
percent in 2013/14. In all three years of analysis, the difference between the two 
groups is statistically significant at the 95 percent confidence level with statistical 
tests based on the standard errors that take clustering into account. 

We also examined whether the impact of the intervention on AP participation 
differed by student subgroups (free and reduced-price lunch status, CTE 
concentrator status) and by school subgroup (rural locale). There are no statistically 
significant differences in the impact of the intervention on AP participation for any 
of these subgroups. 
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Predicted probability of enrolling in an AP course by the end of grade 12 
for students in the comparison and Consortium groups, SYs2012/13, 
2013/14, and 2014/15. 



(2 years exposure) (3 years exposure) (4 years exposure) 


Notes: Resultsare from a two-level hierarchical generalized linearmodel that 
accommodates for clustering of students within schools. The model is used to calculate 
each individual student's probability of enrolling in an AP course using higher ind ivid ua I 
characteristics, and then generating group-level means forthe comparison and 
Consortium groupsusing the intervention variable. The coefficient on the intervention 
variable is 0.826 (p =0.033) in SY 2012/13 ,1.204 (p <0.001) in SY 2013/14, and 0.817 
(p <0.001) in SY 2014/15 . 

Coefficient (of estimated intention-to -treat impact) is statistic a lly significant at the 95 
percent confidence level using a two-tailed test. 


Advanced Placement performance 

Next we examine the impact of the intervention on AP exam scores, an indicator of 
students’ AP performance. This was a dichotomous variable for whether the student 
earned a score of 3 or higher on one or more AP exams at any time during the 
intervention period (l=yes, 0=no). 9 


9 Most state colleges in Tennessee will grant students college credit for AP courses if they score 
at least 3 on the associated AP exam, although some colleges may require a higher score. 
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The exploratory analysis for this outcome includes students from the grade 11 
cohort who had two years of exposure to the intervention at the time of the outcome 
(2012/13); the confirmatory analysis uses students from the grade 10 cohort who 
had three years of exposure to the intervention at the time of the outcome (2013/14). 
During the extension of the evaluation period, an additional year of data is added for 
the grade 9 cohort in 2014/15, who had exposure to the intervention for a full four 
years. 

We find that the Niswonger Foundation i3 grant does have a statistically significant 
impact on performance on AP exams in 2013/14 and 2014/15. While the predicted 
probability of earning a score of 3 or higher remains constant around 8.7 percentage 
points for the comparison group, the predicted probability for the Consortium group 
is 10.3 percent in 2013/14 and 11.1 percent in 2014/15 (figure 5). 


Figure 5. 


100 . 0 % 


75.0% 


50.0% 


25.0% 


Predicted probability of earning an AP exam score of 3 or higher by the 
end of grade 12 forstudentsin the comparison and Consortium groups, 
SYs 2012/13, 2013/14, and 2014/15. 

B Comparison Consortium 


8.7% 9.6% 8.8% 10 - 3 %* 8.7% 11,1% * 



2012/13 2013/14 2014/15 

(2 years exposure) (3 years exposure) (4 years exposure) 


Notes: Resultsare from a two-level hierarchical generalized linearmodel that 
accommodates for clustering of students within schools. The model is used to calculate 
each individual student's probability of earning a score of 3 orhig heron an APexam using 
his/ her individual characteristics, and then generating group-level means for the 
comparison and Consortium groupsusing the intervention variable. The coefficient on the 
intervention variable is 0.098 (p =0.726) in SY 2012/13, 0.938 (p =0.013) in SY 2013/14, and 
0.857 (p =0.016) in SY 2014/15. 
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We find also that for the subsample of students who enrolled in an AP course, the 
likelihood of earning a score of 3 or higher on an AP exam is greater in the 
Consortium schools. For students who enrolled in an AP course, the predicted 
probability of earning a score of 3 or higher on an AP exam is 40.7 percent in the 
Consortium group, compared with 37.2 percent in the comparison group (a 
difference of 3.5 percentage points). This means that even though overall enrollment 
in AP courses is increasing at a faster rate in the Consortium schools relative to the 
comparison schools (as shown in research question 2 on AP participation), the pass 
rates on AP exam are still higher in the Consortium schools relative to the 
comparison schools. This suggests that the Consortium is having a positive effect 
both on the quantity of students enrolled in AP courses and on the quality of 
performance in those courses as demonstrated by performance on AP exams. 

For the exploratory analysis of students with two years of exposure to the 
intervention, there is no impact of the Niswonger Foundation i3 grant on students’ 
AP performance. In SY 2012/13, the predicted probability that a student will achieve 
a score of 3 or higher on an AP exam is 8.7 percent for students in the comparison 
schools and 9.6 percent for students in the Consortium schools (figure 5). 

The difference between the two groups is not statistically significant at the 95 
percent confidence level with statistical tests based on the standard errors that take 
clustering into account. 

College enrollment 

The next outcome examined is the impact of the intervention on whether the student 
enrolled in a postsecondary institution after high school, an indicator of students’ 
college enrollment. This was a dichotomous variable for whether the student enrolled 
in a postsecondary institution by the fall semester following the end of grade 12 
(l=yes, 0=no). This includes two-year colleges, four-year colleges, and Tennessee 
Colleges of Applied Technology. 

There are two exploratory analyses to examine the impact of the intervention on 
college enrollment (1) for students in the grade 12 cohort with one year of exposure 
to the intervention at the time of the outcome (Fall 2012), and (2) for students in the 
grade 11 cohort with two years of exposure to the intervention (Fall 2013). The 
confirmatory analysis examines the impact of the intervention on students in the 
grade 10 cohort with three years of exposure to the intervention. During the 
extension of the evaluation period, an additional year of data is added for the grade 
9 cohort in Fall 2015, who had exposure to the intervention for a full four years. 
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For all analyses, the sample is limited to students who remained in a Consortium or a 
comparison school from the beginning of the grant until the end of grade 12, as we 
are otherwise unable to distinguish between students who did not enroll in college 
and students who were lost to attrition (e.g., students who left the public school 
system in Tennessee, such as by moving out of state). 10 

For the confirmatory analysis of students with three years of exposure to the 
intervention, the Niswonger Foundation i3 grant had a statistically significant impact 
on college enrollment. The predicted probability of enrolling in college by the fall 
semester after the end of grade 12 was 57.7 percent for the Consortium group, 
compared with 54.3 percent for the comparison group, a difference of 3.4 percentage 
points (figure 6). For the additional year of data in Fall 2015, there is a similar 
difference in the predicted probability of college enrollment between the two groups, 
but this difference is no longer statistically significant. 

For the exploratory analysis of students with one year of exposure to the 
intervention, the Niswonger Foundation i3 grant had a statistically significant impact 
on college enrollment. The comparison group probability of 55.5 percent represents, 
for students in the sample, the probability that a student from a comparison school 
will enroll in a postsecondary institution by the Fall semester after the end of grade 
12 (figure 6). The predicted probability for students in the Consortium schools is 
60.1 percent, a difference of approximately 4.5 percentage points. For the 
exploratory analysis of students with two years of exposure to the intervention, 
however, there is no impact of the Niswonger Foundation i3 grant on students’ 
college enrollment. The predicted probability of enrolling in a postsecondary 
institution is 56.3 percent in the comparison schools and 58.6 percent in the 
Consortium schools. This difference of 2.3 percentage points is not statistically 
significant. 

When comparing the results from Fall 2012 and Fall 2013, it appears that the college- 
going rate in the Consortium has declined from 60.1 percent to 58.6 percent. The two 
years are not directly comparable, however, due to differences between the samples 
of the two cohorts. All analyses exclude students who are missing data on the 
pretest. The pretest for the grade 10 cohort is the PLAN, which was missing for 10 
percent of students in the comparison group and 12 percent of students in the 
Consortium group in Fall 2013 (Table 4 4). The pretest for the grade 11 cohort is the 
ACT, which had a much higher rate of missing data, particularly for the Consortium 
group. The percentage of the analysis sample with missing pretest data for this 
cohort was 13.9 percent for the comparison group and 22.9 percent for the 


10 Students who do not graduate from high school due specifically to dropout are included in 
the analysis sample and coded as not enrolled in college. 
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Consortium group. When the evaluation team spoke with staff in the schools and 
districts with high rates of missing data on the ACT pretest, we learned that several 
Consortium schools made the ACT optional in 2010/11 (the pretest year for this 
group) even though all students in grade 11 should have been tested. The ACT is 
required for admission to many colleges, so students who voluntarily take this test 
are probably more likely to be planning on going to college than students who opt 
out. The unadjusted college-going rate including students with missing pretest data 
is 53.3 percent for the Consortium schools in Fall 2012. This rate increases slightly 
to 55.4 percent in Fall 2013. 

Figure 6. Predicted probability of enrolling in college by the fall semester after the 
end of grade 12 forstudentsin the comparison and Consortium groups, 

Fall 2013 through Fall 2015. 



Fall 2012 Fall 2013 Fall 2014 Fall 2015 

(1 year exposure) (2 years exposure) (3 years exposure) (4 years exposure) 


Notes: Resultsare from a two-level hierarchical generalized linearmodel that 
accommodates for clustering of students within schools. The model is used to calculate 
each individual student's probability of enrolling in an AP course using higher ind ivid ua I 
characteristics, and then generating group-level meansforthe comparison and 
Consortium groupsusing the intervention variable. The coefficient on the intervention 
variable is 0.261 (p= 0.028) in Fall 2012, 0.005 (p= 0.928) in Fall 2013, 0.133 (p=0.022) in Fall 
2014, and 0.089 (p =0.120) in Fall 2015. 

Coefficient (of estimated intention-to -treat impact) is statistic a lly significant at the 95 
percent confidence level using a two-tailed test. 


It is also Important to note that the unadjusted college-going rate is similar for the 
comparison schools and the Consortium schools in Fall 2012 when the analysis 
sample includes students with missing pretests. The unadjusted college-going rates 
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for the analysis sample including students with missing pretests is 50.9 percent for 
the comparison group and 53.3 for the Consortium group. This difference of 2.4 
percentage points between the two groups is not statistically significant. These 
findings suggest that the differences in the predicted probabilities of enrollment in 
college for the comparison and Consortium groups in Fall 2012 (shown in table 4) are 
likely attributable to differences in the sample due to missing pretest data. 

Table 4. Percentage of ana lysis sample with pretestdata and missing pretestdata, 
and unadjusted college-going ratesforeach in the comparison group 
and Consortium group, Fall 2012 and Fall 2013. 



Comparison 

Consortium 


Fall 

2012 

Fall 

2013 

Fall 

2012 

Fall 

2013 

Percent of analysis sample with pretestdata 

86.1% 

89.7% 

77.1% 

88.1% 

Percent of analysis sample missing pretestdata 

13.9% 

10.3% 

22.9% 

12.0% 


Unadjusted colleae-qoinq rates for: 

Analysis sample including students with missing 
pretestdata 

50.9% 

54.0% 

53.3% 

55.4% 

Analysis sample omitting students with missing pretest 
data 

55.6% 

56.3% 

60.7% 

58.6% 


We also examined whether the impact of the intervention on college enrollment 
differed by student subgroups (free and reduced-price lunch status, CTE 
concentrator status) and school subgroup (rural locale). There are no statistically 
significant differences in the impact of the intervention by free and reduced-price 
lunch status or rural locale. There were significant differences for CTE concentrators, 
however. Overall, CTE concentrators enroll in college at lower rates than non-CTE 
concentrators. Yet we find that the intervention has a greater impact for CTE 
concentrators. As a result, the gap in college enrollment between CTE concentrators 
and non-concentrators is smaller in the Consortium group relative to the comparison 
group. The probability of enrolling in college for students in the Consortium group is 
52.7 percent for CTE concentrators and 64.0 percent for non-concentrators, a gap of 
11.3 percentage points. In the comparison group, the probability of enrolling in 
college is 44.6 percent for CTE concentrators and 63.8 percent for non-concentrators, 
a gap of 19.2 percentage points. 
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College persistence 

The last outcome examined is the Impact of the intervention on whether the student 
enrolled in a postsecondary institution in both the fall semester immediately after 
high school and in the fall semester one year after high school, an indicator of 
students’ college persistence. This was a dichotomous variable for whether the 
student enrolled in a postsecondary institution for two consecutive fall semesters 
after grade 12 (l=yes, 0=no). This includes in-state colleges and Tennessee Colleges 
of Applied Technology, but excludes students who attended an out-of-state college 
because these data were not available for the persistence outcome. 

The confirmatory analysis examines the college persistence outcome in Fall 2014 for 
students with two years of exposure to the intervention. During the extension of the 
evaluation period, an additional year of data is added for the grade 10 cohort in Fall 
2015, who had exposure to the intervention for three years. 

Similar to the analyses for the college enrollment outcome, the sample is limited to 
students who remained in a Consortium or a comparison school from the beginning 
of the grant until the end of grade 12, as we are otherwise unable to distinguish 
between students who did not enroll in college and students who were lost to 
attrition (e.g., students who left the public school system in Tennessee, such as by 
moving out of state). 11 

The Niswonger Foundation i3 grant had a statistically significant impact on college 
persistence for students with two years and three years of exposure to the 
intervention. The predicted probability of enrolling and persisting in college for two 
consecutive fall semesters after grade 12 was approximately 3 percentage points 
higher for the Consortium group relative to the comparison group in both years 
(figure 7). These difference between the two groups are statistically significant at the 
95 percent confidence level with statistical tests based on the standard errors that 
take clustering into account. 


11 Students who do not graduate from high school due specifically to dropout are included in 
the analysis sample and coded as not enrolled in college. 
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Figure 7. Predicted probability of persisting in college fortwo consecutive fall 

semesters after the end of grade 12 forstudentsin the comparison and 
Consortium groups, Fall 2014 and Fall 2015. 



(2 years exposure) (3 years exposure) 


Notes: Resultsare from a two-level hierarchical generalized linearmodel that 
accommodates for clustering of students within schools. The model is used to calculate 
each individual student's probability of enrolling in an AP course using higher ind ivid ua I 
characteristics, and then generating group-level means forthe comparison and 
Consortium groupsusing the intervention variable. The coefficient on the intervention 
variable is 0.130 (p =0.047) in Fall 2014 and 0.133 (p =0.022)) in Fall 2015. 

* Coefficient (of estimated intention-to -treat impact) is statistic ally significant at the 95 
percent confidence level using a two-tailed test. 


Limitations 

This study has several limitations relating to the generaliz ability of the findings to 
other settings and contexts. One is that all of the school systems participating in the 
Consortium had relationships with the Niswonger Foundation that had been 
established over many years prior to the grant. The Niswonger Foundation was 
established in 2001 with a mission to “create opportunities for individual and 
community growth through education” in Northeast Tennessee. In addition to 
providing funding for education, it also has operated its own programs and 
developed close working partnerships with schools in the region over this time. 
Foundation staff members regularly travel to local communities and work hand-in- 
hand with school leaders to identify specific needs, brainstorm solutions, and then 
provide training, resources, staffing, and materials needed to make improvements in 
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schools. It would likely be much more challenging to try to replicate the activities of 
the Consortium with another group of schools where the same level of trust and 
collaboration had not been previously established. 

In addition, the Consortium is limited to the schools that volunteered to participate. 
They may differ from the broader population of Tennessee high schools and may be 
more willing to embrace educational reform initiatives. For these reasons, the 
findings are limited to the study schools and are not generalizable beyond the 
sample. 

Another limitation is the use of quasi-experimental design to identify the comparison 
group. Propensity score matching was used to select schools that were similar to the 
Consortium in terms of demographic characteristics of the student population, test 
scores of the student population, school resources, and community characteristics. 
But it is possible that the Consortium and comparison groups differ on other 
characteristics not observed, such as school culture. Any unobserved differences may 
contribute to differences in student outcomes between the two groups. 

The geographic proximity of the Consortium schools may be a potential confound, as 
well. All of the Consortium schools are located in adjacent counties in the First 
Congressional District of Tennessee, while the comparison schools are located 
throughout the state. This may lead to regional differences that contribute to 
differences in student outcomes. 
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findings fidelity of Implementation 


This section of the report reviews the findings from the fidelity of implementation 
analysis conducted between SYs 2011/12 and 2014/15. Because the first year of the 
grant (SY 2010/11) was used primarily for planning and building necessary program 
infrastructure, findings from this baseline year are not reported. The fidelity of 
implementation analysis examines the Consortium’s efforts to successfully 
implement strategies to improve college and career readiness among high school 
students. 

Overall fidelity of implementation for key 
program components 

Table 5 summarizes for each of the four years of full implementation the fidelity of 
that implementation for each of the Consortium’s key program components. It 
includes an overall assessment (“Met with fidelity?”) and the number of indicators 
used to measure fidelity within each component. 

The change in Consortium leadership in Summer 2013 resulted in changes in the 
organization of the program, adjustments to some activities, and revisions in the 
structure of each program component (see Appendix C, table 11). Even with these 
changes, however, the components remained consistent thematically throughout the 
three school years evaluated, even if their specific indicators changed over time. This 
means that any large differences in fidelity scores from one year to the next are 
likely attributable to the effect of those changes on the way the intervention was 
implemented over time. Smaller differences may be similarly due to differences over 
time in program implementation, or to changes in the way fidelity is measured. One 
indicator— measuring the Consortium’s collaboration with Local Education Agency 
(LEA) partners— was dropped completely after SY 2012/13, so it is measured only for 
the first two years of implementation. Most of its indicators were moved to other 
components. 

The pattern of components meeting or not meeting the fidelity benchmark (i.e., an 
average score of 2.5 or higher across its indicators) is mostly consistent across 
evaluation years. The three components Management and Communication, Resources 
and Services, and LEA Partners all met the benchmark in every year they were 
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measured. In contrast, the components College and Career Counselors Team, 
Learning Resources Team (distance/online technology), and Learning Resources 
Team (college-level courses) did not meet the benchmark in any of the four years. 
Looking simply at the yes/no fidelity assessment, however, obscures the fact that the 
Learning Resources Team component for distance and online technology nearly met 
fidelity in SYs 2012/13 and 2013/14 (as described below) before falling just below 
moderate fidelity in SY 2014/15 Likewise, the Learning Resources Team component 
for quality of instruction met moderate fidelity in SYs 2013/14 and 2014/15, having 
been implemented with fidelity the previous two school years. 

We describe the fidelity of implementation of each of these key components in more 
detail below. 


Table 5. Summary of key program component fidelity, SYs2011/12to 2012/13. 



2011/12 

2012/13 

2013/14 

2014/15 

Component 

Met with 
fidelity? 

#of 

indicators 

Met with 
fidelity? 

#of 

indicators 

Met with 

fidelity? 

#of 

indie atois 

Met with 

fidelitv? 

#of 

indicators 

1) i3 leadership managementand 
communication 

Yes 

8 

Yes 

8 

Yes 

8 

Yes 

8 

2) College and Career Counselors 
learn 

No 

8 

No 

8 

No 

6 

No 

6 

3) Learning Resources learn: quality of 
instruction 

Yes 

5 

Yes 

5 

No 

4 

No 

4 

4) Learning Resources learn: 
distance/online technology 

No 

7 

No 

7 

No 

7 

No 

7 

5) Learning Resources learn: college- 
level courses 

No 

3 

No 

3 

No 

5 

No 

5 

6) Resources and services 

Yes 

4 

Yes 

4 

Yes 

4 

Yes 

4 

7) LEA partners 

Yes 

6 

Yes 

6 

- 

- 

- 

- 


(1) i3 leadership provides managementand 
communication 

Measurements of this program component’s indicators remained mostly consistent 
throughout the four evaluation years. The component is composed of eight 
indicators; however, after SY 2012/13, two indicators— frequency of Consortium 
leadership’s contact with the evaluator, and level of staff participation in weekly 
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meetings— were replaced by two indicators measuring how often schools shared 
resources, and participation in course review meetings by school system 
representatives. 

This component met the fidelity benchmark in all four evaluation years (figure 8). 
Over the years, the Consortium leadership typically earned high fidelity scores for 
their ability to convene the Advisory Board regularly, share information with all staff 
members, and communicate with school systems and outside stakeholders. In 
addition, Consortium staff members earned a high fidelity rating for completing 
professional development activities (e.g., attending conferences or training sessions) 
and conducting outreach to legislative bodies in SYs 2013/14 and 2014/15. 


Figure 8. Fidelity of implementation scoresforthe component "i3 leadership 
providesmanagement and communication/' byyear. 



Advisory Board attendance at the quarterly meetings is the only area that declined 
over time. This indicator shifted from a high fidelity rating in SY 2011/12 to low 
fidelity in SY 2012/13 and beyond. Changes in the composition of the Advisory 
Board and the physical geography of the members likely explain this shift. In SYs 
2011/12 and 2012/13, the Board was small and composed primarily of individuals 
located near the Board’s meetings in Greenville, Tennessee. Beginning in SY 2013/14, 
the Leadership Team expanded the Advisory Board to include a representative from 
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every district in the Consortium. As a result, the Board became much larger and more 
diverse, so achieving high meeting attendance became more difficult. 


(2) College and CareerCounseloisleam 
promotes a college-going culture 


The component measuring the College and Career Counselors Team’s efforts to 
promote a college-going culture had eight indicators in SYs 2011/12 and 2012/13, 
and six indicators in SYs 2013/14 and 2014/15. Three indicators measuring the 
district’s counselor-to-school ratio, number of group or class presentations, and 
creation of informational materials were eliminated after SY 2012/13. A new 
indicator was added in SY 2013/14 for collaboration with other counselors and 
educators in the school-wide integration of the guidance services. There were also 
some smaller changes over time to the indicators for the core activities of the 
counselors and the college visits they organized. 

The results in figure 9 show that the component did not reach fidelity in any of the 
four evaluation years. Based on counselor logs, the indicators for setting up college 
visits received low fidelity ratings in all four evaluation years. Of the 29 Consortium 
schools evaluated, counselors from 6 schools set up onsite college visits during SY 
2011/12 and this number did not change in SY 2014/15. No virtual college visits 
were set up during SYs 2013/14 or 2014/15 (these virtual visits could count toward 
fidelity scoring in these years only). The indicator measuring time spent on 
individual college and career counseling scored high fidelity in SYs 2011/12 and 
2012/13, but dropped to low fidelity in SY 2013/14 before increasing to moderate 
fidelity in SY 2014/15. 

Counselors also received low fidelity scores for indicators measuring their level of 
collaborative effort in SY 2013/14. Out of the 29 Consortium schools, 14 schools had 
CACCs who met with educators, community members, and parents about future 
planning for students, which met low fidelity for the indicator. The College and 
Career Counselors Team received higher fidelity ratings for their collaboration with 
other schools in SY 2014/15. For each Consortium school, if counselors averaged one 
communication per month with other counselors, the school was scored as meeting 
high fidelity. Out of the 29 Consortium schools, 21 met this criterion, four schools 
shy of high implementation. 
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Figure 9. Fidelity of implementation scoresforthe component "College and Career 
Counselors Team promotesa college-going culture," by year. 



(3) beaming Resources Team increases 
quality of instruction 

The fidelity score for the component measuring the Learning Resources Team’s 
efforts to increase the quality of instruction consisted of five indicators in SYs 
2011/12 and 2012/13, and four indicators in SYs 2013/14 and 2014/15. None of the 
indicators within this component was measured consistently in all four years of the 
evaluation, making it difficult to compare fidelity scores over time. Between SYs 
2012/13 and 2013/14, the item measuring the district’s instructional specialists-to- 
school ratio was eliminated, and the rest of the indicators were moved to other 
components. 12 Four new indicators for SYs 2013/14 and 2014/15 measure 


12 For example, the indicators measuring the quality of online instruction and the assessment 
of online courses for possible adoption were moved in SYs 2013/14 and 2014/15 to 
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coordination efforts to offer professional development events for teachers, online 
communities where teachers could collaborate, training for online teachers, and 
STEM instruction assistance, respectively. Although this component consistently 
focuses on instructional quality across the four evaluation years, comparisons 
between school years should be interpreted with caution, since the composition of 
indicators changed substantially over the period. 

Overall, this component met fidelity in SYs 2011/12 and 2012/13, but not in SY 
2013/14 and SY 2014/15 (figure 10). During the first two evaluation years, the 
indicator measuring support provided to teachers of AP, online, and distance 
learning courses met low fidelity. The indicator for instructional specialists-to-school 
district ratio met moderate fidelity. The remaining three indicators for organizing 
events to share best practices, evaluating online courses for possible adoption, and 
creating a Consortium-wide course directory met high fidelity during SYs 2011/12 
and 2012/13. 

Among the component’s four new indicators employed during SYs 2013/14 and 
2014/15, the indicator measuring online collaborative professional communities was 
scored as low fidelity in both years because a planned online collaborative site for 
professional development was not created. In addition, the Consortium had a low 
fidelity rating for providing outreach efforts to mathematics, science, and technology 
teachers through the STEM professional development events and/or individual 
outreach efforts in SY 2014/15 after earning a moderate fidelity ranking in SY 
2013/14. The Consortium’s online Coordinator was still able to provide training to 
online teachers each semester, however, which earned a high fidelity rating for that 
indicator in SYs 2013/14 and 2014/15. And the Consortium staff provided 14 
professional development events during Summer 2015. In SYs 2013/14 and 2014/15, 
since at least one teacher from almost all 29 Consortium schools attended a 
professional development event, that indicator earned high fidelity. 


component 4: “Learning Resources Team increases access to courses through distance and 
online technology.” 
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Figure 10. Fidelity of implementation scoresforthe component "Learning Resources 
Team increases quality of instruction," by year. 


3.0 



(4) Learning Resources learn increases 
access to courses through distance and 
online technology 

The component measuring the Consortium’s efforts to expand access to online and 
distance learning courses had seven indicators, but they were not the same 
indicators in all four years. Five of the indicators in this component in SYs 2011/12 
and 2012/13 were modified slightly and moved to the “Resources and Services” 
component in SYs 2013/14 and 2014/15. The five were monitor technology in 
schools and assist in using technology, identify and acquire technology to support 
distance and online learning, ensure reliability of equipment, create a distance and 
online handbook, and maintain a Consortium-wide online learning center. Two 
indicators in the component “Learning Resources Team: Quality of Instruction” in 
SYs 2011/12 and 2012/13 were moved from there to this component in SYs 2013/14 
and 2014/15: recruit/retain and supervise online teachers, and assist/train school 
personnel in using technology. In addition, three new indicators were added in SYs 
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2013/14 and 2014/15 for coordinating online course development, coordinating 
online course evaluation, and ensuring schools use ACT Prep online software. 

As shown in figure 11, this component did not meet fidelity in any evaluation year. In 
SY 2011/12, no fidelity score was calculated for three out of the seven indicators due 
to a lack of data, so the overall fidelity of the component for SY 2011/12 was based 
on an average of the other four indicators only. The lack of fidelity in subsequent 
years is attributable partially to the distance learning enrollment indicator. Among 
the 29 Consortium schools, actual enrollment versus the projected goal earned a low 
fidelity rating for distance learning courses between SYs 2012/13 and 2014/15. This 
may be partially the result of overly aggressive yearly goals under the i3 grant. The 
projected goal for distance learning enrollments was more than triple: from 377 
students in SY 2010/11 to 1,497 students in SY 2014/15. In SY 2014/15, fewer than 
half of the courses identified in the supply and demand reviewing meeting were 
made online. In addition, less than 50 percent of the Consortium schools had a 
student who used the online ACT prep software. For these reasons, both items were 
scored low fidelity. 

Despite these lower ratings, two items did meet high fidelity in SYs 2013/14 and 
2014/15. Enrollment in online courses exceeded the projected goal, and the online 
coordinator reviewed at least 75 percent of courses evaluated to ensure they met 
Tennessee standards in these years. 
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Figure 11. Fidelity of implementation scoresforthe component "Learning Resources 
Team increasesaccessto coursesthrough distance and online 
technology," by year. 



(5) Learning Resources Team expands 
opportunities for college- level couvses 

The component measuring the Consortium’s efforts to expand high school students’ 
access to college-level courses was composed of three indicators in SYs 2011/12 and 
2012/13, then underwent several changes. An indicator measuring the expansion of 
dual enrollment courses was used consistently throughout all three evaluation years. 
Indicators measuring activities to review course supply and demand, and the delivery 
of professional development events to Consortium staff by college instructors were 
removed for SYs 2013/14 and 2014/15. 

In their place, several indicators measuring college-credit course implementation 
were added: for marketing to promote the value of AP courses, expansion of AP 
course enrollments, and coordination efforts between high schools and 
postsecondary institutions to implement dual enrollment courses. A fourth indicator 
in SYs 2013/14 and 2014/15— assist in the implementation of AP programs— was 


48 


CNA 


similar to an indicator from SYs 2011/12 and 2012/13 in the component “Learning 
Resources Team increases quality of instruction.” 

As shown in figure 12, this component did not meet fidelity in any of the evaluation 
years. The indicator measuring actual enrollment versus projected enrollment goals 
for dual enrollment courses earned a moderate fidelity rating in SY 2011/12 and low 
fidelity ratings between SYs 2012/13 and 2014/15. Professional development events 
delivered to Consortium staff members by college instructors also earned low fidelity 
during SYs 2011/12 and 2012/13 before the indicator was dropped. 

Figure 12. Fidelity of implementation scoresforthe component “Learning Resources 
Team expandsopportunitiesforcollege-level courses/' by year. 


3.0 


2.5 


2.0 
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0.5 


0.0 



SY 2011/12 SY 2012/13 SY 2013/14 SY 2014/15 

— Fidelity Met 


The indicators measuring AP course enrollment and marketing efforts also earned 
low fidelity ratings in SYs 2013/14 and 2014/15. For the indicator measuring the 
expansion of AP courses, actual total enrollment versus the projected enrollment 
goal for SYs 2013/14 and 2014/15 fell below a 20 percent gain, the benchmark for 
moderate implementation fidelity. Despite the low fidelity rating for this indicator, 
among the seven Consortium schools that had not offered any AP courses at the 
beginning of the grant, five schools offered an AP course in SY 2014/15, which 
resulted in a high fidelity rating for the indicator measuring AP course 
implementation. 
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The Consortium also received a high fidelity rating on a new indicator in SYs 
2013/14 and 2014/15 on coordination efforts, for maintaining good working 
relationships with the postsecondary institutions offering dual enrollment courses to 
Consortium schools. 


(6) Resources and sewices provide 
infrastructure to expand and sustain progvam 
capacity 

The fidelity score for the component measuring the maintenance of resources and 
services needed to expand and sustain the program consisted of four indicators 
across all years of the evaluation, although the definitions of three of them changed 
over time. Only an indicator measuring the purchase and maintenance of distance 
and online learning technology was used in all four evaluation years. 

One indicator measuring the availability of Internet access to support distance and 
online learning, and another indicator for the maintenance of a distance learning 
server were removed after SY 2012/13. Two indicators measuring the maintenance of 
an online course tracking system (openSIS) and a learning center website were added 
in their place beginning in SY 2013/14. An indicator about the creation and 
maintenance of a Consortium-wide course directory that had been included in the 
component “Learning Resources Team increases quality of instruction” in SYs 
2011/12 and 2012/13 was moved to this component for SYs 2013/14 and 2014/15. 
Conversely, another indicator in this component in SYs 2011/12 and 2012/13— 
providing teacher and staff training— was moved to “Learning Resources Team 
increases access to courses” for SYs 2013/14 and 2014/15. 

Fidelity was met in all four evaluation years, as shown in figure 13. Each indicator in 
every evaluation year scored at high fidelity, except two. The course directory was 
not created for SYs 2013/14 and 2014/15, which resulted in a low fidelity rating; and 
the purchase and maintenance indicator earned moderate fidelity in SY 2011/12 
because some schools did not yet have a distance learning lab or computers available 
for distance and online learning. 


50 


CNA 


Figure 13. Fidelity of implementation scoresforthe component "Resourcesand 

services provide infrastructure to expand and sustain program capacity," 
by year. 



(7) LEA partners shane mission and resources, 
adopt practices in their schools* and 
encourage cultural change 


The last component, measuring the collaborative efforts of the Local Education 
Agency partners, comprised six indicators in SYs 2011/12 and 2012/13. This 
component was eliminated for SYs 2013/14 and 2014/15; however, most of its 
indicators were moved to other components. 13 The “LEA partners” component met 


13 An indicator measuring collaboration between the Consortium schools and staff was split for 
SY 2013/14 into three different indicators by course type (AP, online, and distance learning) 
and distributed among the various Learning Resources Team components. Other indicators 
measuring the expansion of AP course enrollments, collaboration among Consortium schools, 
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fidelity in both evaluation years (figure 14). The indicator measuring whether l. F. A s 
picked up recurring costs consistently received a low fidelity rating, but the other 
three indicators had moderate or high fidelity ratings in both years. 

Figure 14. Fidelity of implementation sc ores for the component "LEA partnersshare 
mission and resources, adopt practices in their schools, and encourage 
cultural change," by year. 



Overall assessment of implementation 

Taken together, the scoring data reveal three areas of the i3 grant that consistently 
received high ratings for implementation fidelity throughout the evaluation period— 
Consortium leadership, resources and services to provide program infrastructure, 
and collaboration with LEA partners. 


and participation in Consortium professional development events also were moved to other 
components for SY 2013/14. 
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Although the LEA partners component was eliminated before SY 2013/14, several of 
its indicators measuring collaboration between the Consortium staff and schools that 
were moved to other components still earned moderate or high fidelity ratings in SYs 
2013/14 and 2014/15, such as assistance with the implementation of AP courses. 

Even though there was only partial implementation on several other components, the 
Consortium still had statistically significant impacts on ACT scores, AP course 
taking, and college enrollment. If all components had been met with high fidelity, 
however, it is possible that the intervention might have led to larger or more 
meaningful impacts. There was also variation in the fidelity of implementation by 
school, which may have contributed to differences in the impact of the intervention 
from one school to the next. 

It is important to note that most of the program components that did not meet the 
benchmark for implementation with fidelity were very close to meeting this 
threshold. The numeric fidelity scores are 2.0 or higher on a 3-point scale (equivalent 
to moderate fidelity of implementation) for all components and years with a few 
exceptions. The first is that the College and Career Counselors Team component 
received an overall weighted average score of 1.5 in SY 2013/14 and 1.7 in SY 
2014/15. This is attributed to the low fidelity score assigned to the indicators 
measuring the number of college visits arranged by counselors. There were slight 
differences in fidelity ratings for specific indicators between SYs 2013/14 and 
2014/15. In SY 2013/14, the amount of time counselors spent with students on 
education, career, or life planning earned a low fidelity rating. In SY 2014/15, this 
indicator earned a medium fidelity ranking but the indicator measuring counselors’ 
efforts to collaborate with educators, community members, and parents to assist 
students with educational, career, and life planning changed to a low fidelity ranking 
from a medium fidelity ranking the previous school year. These areas in each school 
year made up half of the component’s indicators, which led to low overall fidelity 
ratings for this component in each year. 

Another exception is the Learning Resources Team component for the expansion of 
college-level courses, which received a score of 1.5 in SY 2012/13 and 1.8 in SYs 
2013/14 and 2014/15. The primary reasons for the first low rating are that the 
Consortium did not conduct a supply and demand review, and limited professional 
development was provided to Consortium staff by partner colleges. Then the 
indicators within this component changed in SY 2013/14. The primary reasons for 
the second low rating are the lack of progress toward the Consortium’s goals for 
student-level participation in dual enrollment and AP courses, and lack of activities 
to promote AP courses within Consortium schools. Lastly, the Learning Resources 
Team component to increase access to online and distance learning courses earned a 
score of 1.9 in SY 2014/15. Small changes in outreach activities would have moved 
this component to moderate fidelity. 
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Another important consideration is that there may be good reasons why some 
components were implemented differently than initially intended, so a low fidelity 
rating is not necessarily an indicator of poor performance. In some cases, fidelity 
may have declined because strategic decisions were made to focus efforts in a 
different direction. For example, distance learning courses did not expand as rapidly 
as originally intended because there were unanticipated challenges (e.g., differences 
in school calendars and bell schedules across districts) that made it difficult to 
implement these courses. As a result, the Consortium staff shifted their emphasis to 
increasing enrollments in online courses, because there were fewer barriers to that 
delivery method. In other cases, low fidelity may occur for unintentional reasons; for 
example, if Consortium schools implemented only some components because they 
didn’t have the capacity or resources to participate in all of them. 

Regardless, it is important to report on these indicators to show how the 
Consortium’s activities were actually implemented, so that if the intervention were to 
be replicated elsewhere, there would be a better understanding of what the 
intervention looked like under the Niswonger Foundation i3 grant. 
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Conclusions 


Summary of results 

As summarized in table 6 below, we find that the Niswonger Foundation i3 grant has 
statistically significant effects on college readiness (after three years of exposure to 
the intervention), AP participation (after two, three, and four years of exposure to the 
intervention), AP performance (after three and four years of exposure to the 
intervention), college enrollment (after three years of exposure to the intervention), 
and college persistence (after two and three years of exposure to the intervention). 
There is also a positive effect on college enrollment (after one year of exposure to the 
intervention), although this is likely attributable to differences in the sample due to 
missing data. 

These findings may demonstrate evidence that the Consortium has made progress 
toward its goals of ensuring that high school students in Northeast Tennessee are 
provided the opportunity to graduate from high school “college or career ready,” and 
of improving the likelihood that students will be successful in college. 

Table 6. Summary ofthe impactsofthe Niswonger Foundation Bgranton student 
outcomes. 





Regression Adjusted Mean or 
Predicted Probability 

Outcome 

Cohort 

Years of 
exposure 

Consortium 

group 

Comparison 

group Difference 

Mean composite ACT 

Grade 11 

2 years 

19.5 

19.5 

0.0 

scores at the end of 
grade 12 (Spring 2014) 

Grade 10 

3 yea is 

19.9 

19.6 

0.3* 

Grade 9 

4 yea is 

19.7 

19.7 

0.0 

Probability of 

Grade 11 

2 years 

25.6% 

22.8% 

2.8%* 

enrollment in an AP 
course by the end of 
grade 12 (SYs2012/13, 
2013/14, and 2014/15) 

Grade 10 

3 yea rs 

27.8% 

21.5% 

6.3%* 

Grade 9 

4 years 

26.4% 

21.9% 

4.5%* 

Probability of earning a 

Grade 11 

2 years 

9.6% 

8.7% 

0.9% 

score of 3orhigheron 

Grade 10 

3 years 

10.3% 

8.8% 

1.5%* 

an APexam by the 
end of grade 12 (SYs 
2013/ 14 and 2014/15) 

Grade 9 

4 years 

11.1% 

8.7% 

2.4%* 
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Regression Adjusted Mean or 
Predicted Probability 



Years of 

Consortium 

Comparison 


Outcome 

Cohort 

exposure 

group 

group Difference 

Robability of 

Grade 12 

1 year 

60.1% 

55.5% 

4.5%* 

enrollment in college 
by the foil semester 
after the end of grade 

12 (Fall 2012, Fall 2014) 

Grade 11 

2 years 

58.6% 

56.3% 

2.3% 

Grade 10 

3 years 

57.7% 

54.3% 

3.4%* 

Grade 9 

4 years 

64.6% 

60.6% 

4.0% 

Robability of 

Grade 11 

2 years 

42.5% 

39.5% 

3.1%* 

persistence in college 
for two consecutive 
foil semesters afberthe 
end of grade 12 (Fall 
2014, Fall 2015) 

Grade 10 

3 years 

40.6% 

37.9% 

2.7%* 


* Coefficient (of estimated intention-to -treat impact) is statistic a lly significant at the 95 
percent confidence level using a two-tailed test. 


The impact evaluation is designed to meet i3 criteria with reservations, the highest 
rating possible for studies with non-experimental designs. The results of this 
evaluation will provide evidence of the effectiveness of the Niswonger Foundation i3 
grant that may be useful for applying for future grants to scale up or expand upon 
the Consortium’s activities. 

Data from the implementation evaluation are helpful for exploring the mechanisms 
through which the intervention achieved its impact. We find that the areas that were 
most consistently implemented as intended throughout the four years of the grant 
were Consortium leadership, resources and services to provide program 
infrastructure, and collaboration with LEA partners. Among the components of the 
intervention that did not meet fidelity, most had fidelity ratings that were very close 
to meeting this threshold. It is also important to consider that this is a new 
intervention and the components had never been implemented together before, so it 
is not surprising that there were some changes from what was originally planned in 
the grant application. 
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Appendix A: Criteria for the NEB 
Evaluation 


This appendix provides additional details on the criteria used to assess the external 
evaluations submitted to the NEi3. 


Criteria for assessing the strength of evidence 

As mentioned in the report, the highest rating that can be received by an i3 
evaluation that uses a quasi-experimental design is “meets i3 criteria with 
reservations.” Below are three criteria that must be met to receive this rating. 

• Criterion 1: The study must establish baseline equivalence in the analysis 
sample. That means that students in the intervention group (those who 
attend a Consortium school) must have similar characteristics to students in 
the comparison group prior to the start of the grant. 

• Criterion 2: The outcome data must meet What Works Clearinghouse (WWC) 
outcome standards. The outcomes are the measures used to examine 
whether the i3 grant had an impact on students. 

o The first requirement is that the outcome measures have face 
validity, which means that they appear to measure the construct of 
interest. For example, a state’s standardized reading test has face 
validity for measuring reading achievement, but it is unlikely to have 
face validity for measuring mathematics achievement. 

o The second requirement is that the outcomes must be able to be 
measured reliably, so that a student’s score on the measure is a good 
representation of the student’s true achievement. 

o The third requirement is that the outcome may not be overaligned 
with the intervention. An overaligned outcome provides an unfair 
advantage to students in the intervention group or the comparison 
group. For example, if students in a reading intervention have a test 
on reading comprehension that covers the same passages they have 
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studied as part of the intervention, while students in the comparison 
group take the same test but have not previously seen the passages. 

o The fourth requirement is that the outcome data must be defined and 
collected consistently between the two groups. For example, a study 
would not meet this requirement if the outcomes for the comparison 
group were collected in the fall semester, while the outcomes for the 
intervention group were collected in the spring semester, giving the 
intervention group an unfair advantage of additional learning time. 

o The fifth requirement is that the outcome data must not be imputed 
for any cases. This means that students’ actual outcomes are used, 
and any students who are missing outcomes are excluded from the 
analysis. 

• Criterion 3: The study must not suffer from a serious confound. A confound 
is an observed component that is part of only one study condition (the 
intervention group or comparison group) and is not part of the intervention 
being evaluated. For example, if all teachers in the intervention group had 
master’s degrees and none of the teachers in the comparison group had a 
master’s degree, it would not be possible to distinguish whether any 
differences in student outcomes were attributable to the presence of more- 
educated teachers or to the intervention itself. 

The impact evaluation for the Niswonger Foundation i3 grant was designed to meet 
all of the NEi3 criteria needed to receive a rating of “meets i3 criteria with 
reservations.” 

Criteria for assessing the quality of 
implementation findings 

The NEi3 team developed its own criteria for assessing the logic model and the plan 
for measuring fidelity of implementation, as described below (Abt Associates, 2012): 

• Criterion 1: Fidelity must be measured separately for each of the key 
components of the intervention. For the Niswonger Foundation i3 grant, the 
evaluation team worked closely with the Consortium Director to identify the 
key components that should be included in the logic model and the fidelity 
measures. 

• Criterion 2: Fidelity measures must include the entire sample participating in 
the intervention. For the Niswonger Foundation i3 grant, this means that the 
data must include all of the original 29 schools in the Consortium. 


58 


CNA 

ANALYSIS A SOLUTIONS 


• Criterion 3: The evaluator must define specific thresholds (represented by 
quantifiable cut-off scores) for determining whether the key components 
were implemented with fidelity for the entire sample. For the Niswonger 
Foundation i3 grant, the evaluation team worked closely with the Consortium 
Director to define the specific threshold for each component. 

• Criterion 4: The evaluator must assess and report whether each of the key 
components was actually implemented with fidelity or not implemented with 
fidelity. For the Niswonger Foundation i3 grant, the evaluation team has been 
collecting data on implementation for each year of the grant and assessing 
whether each component was implemented with fidelity based on the 
thresholds defined in Criterion 3. 

Evaluations that satisfy these criteria will be described as “providing high-quality 
implementation data, performance feedback, and periodic assessments of progress 
for the intervention,” which is one of the requirements for federal grants under the 
Government Performance and Results Act (GPRA). This assessment is used to 
characterize the quality of the implementation study only, and does not contribute to 
the NEi3 rating. 

The implementation evaluation for the Niswonger Foundation i3 grant was designed 
to meet all of the NEi3 criteria for high-quality implementation data. 

Other relevant factors for assessing i3 
evaluations 

The NEi3 team identified six additional factors that provide stakeholders with 
additional information about the quality of the i3 evaluations (Abt Associates, 2012). 
These factors are used to provide additional context only, and do not contribute to 
the NEi3 rating. 

1. Independence: Key aspects of the evaluation must be conducted by an 
independent, third-party evaluator, and the findings of the evaluation are not 
subject to approval by the grantee. 

2. Relevance: The evaluation must be relevant for assessing the impact of the 
intervention by using a representative sample of the population that received 
services, and by ensuring that the evaluation assesses the intervention as it 
was actually implemented. 

3. Measuring the contrast: The evaluation must present differences in outcomes 
for the intervention group relative to a comparison group. It is not sufficient 
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to compare outcomes before and after the intervention only for the 
intervention group. 

4. Statistical power: The NEi3 report will indicate whether the sample size was 
large enough to distinguish a minimum detectable effect on student, teacher, 
or school outcomes. 

5. Internal validity: There are several factors used to determine internal validity, 
as described below. 

a. The intervention and comparison groups must not be subject to any 
confounds that make it difficult to distinguish whether any effects 
are attributable to the intervention or to some other factor. For 
example, if all the intervention students attended one school and all 
of the comparison students attended another single school, any 
differences in student outcomes could be due to the unique 
characteristics of the individual school in each condition rather than 
to the intervention itself. 

b. The person rating or scoring the outcome data must not be aware of 
the intervention status of the sample members from whom the rater 
is collecting data. 

c. The comparison group must not be subject to selection bias, meaning 
that the intervention group and the comparison group cannot be 
inherently different from each other. For example, schools that agree 
to participate in the intervention are likely to differ from schools that 
decline on factors such as their priorities for school improvement. If 
the schools that declined to participate were included in the 
comparison group, the results could be biased, because the 
intervention group would be compared with schools with no interest 
in participating in the intervention. 

6. Scientific process: The evaluation team for each i3 grant was asked to 
prepare an analysis plan prior to conducting the analyses that outlined the 
procedures for defining the sample and any subgroups to be included in the 
analyses, identifying the outcomes for the evaluation, establishing the 
procedures for testing the baseline equivalence between the intervention and 
comparison groups, and the methods for conducting the impact analysis. 
Any changes from the prespecified analysis plan must be documented with 
an explanation. This prevents the evaluators from changing the analysis in an 
effort to obtain more favorable results. 

The evaluation for the Niswonger Foundation i3 grant was designed to satisfy the 
requirements for all of these additional factors for assessing i3 evaluations. 
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Appendix B: Technical Information 
on Propensity Score Matching and 
Statistical Models 


This appendix contains technical details on the procedures for the propensity score 
matching used to identify the comparison schools and the statistical models used to 
estimate the impact of the intervention. 


Propensity score matching 


Twenty-nine (29) high schools in 15 school districts were selected to participate in 
the intervention prior to the award of the grant. This is a convenience sample of 
districts in Northeast Tennessee that had previously worked in a partnership with 
the Niswonger Foundation on other projects. 

A school-level propensity score model was used to identify 29 comparison schools 
among the 288 regular public high schools in Tennessee. A binary logit regression 
model estimates the probability that a school is a member of the Northeast 
Tennessee College and Career Ready Consortium. For each school, the propensity 
score is defined as: 


P(X = 1 ) = 


oP' X 


l + e 


,fi’x 


In the formula, Y represents the outcome variable (which equals 1 if the school is a 
member of the Consortium and 0 otherwise), which represents a probability score. X 
is the set of observed school characteristics representing demographic 
characteristics of the student population, test scores of the student population, 
attendance and graduation rates, school resources, community characteristics, and 
CTE and AP course taking (table 7). P represents the estimated coefficients from the 
regression model. All covariates represent characteristics prior to the intervention, so 
their values are not influenced by treatment status. 
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Table 7. List of variables included in the propensity score model to selectthe 
matched comparison schools. 


Type of Characteristic 


Variables 

Demographic 

C ha rac teristic s of the 
Student Population 

• 

• 

• 

• 

• 

Percent of students who are minorities (races otherthan 

White) 

Percent of students who are male 

Percent of students who receive free or reduced-price lunch 
Percent of students who are Limited English Proficient 

Percent of students with an Individualized Education 

Program 

lest Sc ores of the 
Student Population 

• 

• 

• 

• 

• 

• 

• 

• 

3-yearaverage ACTEnglish score 

3-yearaverage ACT mathematics sc ore 

3-yearaverage ACT reading score 

3-yearaverage ACTscience score 

3-yearaverage algebra end-of-course exam score 
3-yearaverage biology end-of-course exam score 
3-yearaverage English end-of-course exam score 
3-yearaverage history end-of-course exam score 

Attendance and 
Graduation Rates 

• 

• 

Average school attendance rate (percent of days present) 
High school graduation rate (percent of on-time graduates 
with a regular high school diploma) 

School Resources 

• 

• 

• 

Stu d e n t/ te a c h e r ra tio 

Total expenditures per pupil ($) 

School size (enrollment in grades 9-12) 

Community 

Characteristics 

• 

• 

• 

• 

• 

Percent of population with a college degree 

Percent of population below the poverty line 

Population's median annual income 

Rural school locale (l=yes ; 0=no) 

Distance to nearest public college (miles) 

CTE and AP Course 
Taking 

• 

• 

• 

• 

Number of CTE program areas offered by the high school 
Percent of high school graduates with CTE concentrations 
Numberof AP courses offered by the school 

Ratio of enrollment in AP coursesto total enrollment in 
grades 11 and 12 


A propensity score is calculated for each school in the sample by substituting that 
school’s set of observed characteristics (A) and the estimated coefficients (P) from 
the logit regression into the propensity score formula as defined in equation (1). The 
propensity score is therefore the estimated probability that the school was selected 
to receive the intervention. The 29 non-Consortium schools with the highest 
propensity scores are each matched to a Consortium school. Matches are selected 
without replacement using nearest-neighbor matching. 
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Table 8. Comparison of the baseline characteristics of the Consortium schools versus a II otherTennessee schoolsand the 
matched comparison schools from the propensity score model. 


Consortium 

Sc hools All Otherlennessee Sc hools Mate hed Comparison Sc hools 

(N=29) (N=259) (N^29) 


Std. Std. p Std. p 



Mean 

Dev. 

Mean 

Dev. 

Differ. 


value 

Mean 

Dev. 

Differ. 

value 

Student Demographic s 

% Minority 

0.05 

0.05 

0.27 

0.31 

-0.22 

** 

0.01 

0.05 

0.05 

0.00 

0.93 

% M a le 

0.52 

0.02 

0.51 

0.02 

0.01 


0.15 

0.52 

0.02 

0.00 

0.69 

%Fnee or reduced-price lunch 

0.55 

0.18 

0.54 

0.21 

0.01 


0.84 

0.54 

0.14 

0.01 

0.79 

%l_imited English Proficiency 

0.01 

0.02 

0.02 

0.02 

-0.01 

* 

0.01 

0.01 

0.01 

0.00 

0.92 

% Individualized Education Program 

0.07 

0.01 

0.08 

0.01 

-0.01 

** 

0.01 

0.07 

0.01 

0.00 

0.37 

lest Sc ones 

ACTEnglish score (3 yravg) 

20.36 

1.19 

19.67 

2.17 

0.69 


0.09 

20.26 

1.25 

0.10 

0.76 

ACTMathematicsscore (3 yravg) 

19.55 

1.41 

19.00 

1.72 

0.55 


0.10 

19.63 

1.37 

-0.08 

0.82 

ACTReading score (3 yravg) 

21.02 

1.05 

20.09 

2.02 

0.93 

* 

0.02 

20.83 

1.13 

0.19 

0.51 

ACTScience score (3 yravg) 

20.26 

0.89 

19.61 

1.58 

0.65 

* 

0.03 

20.16 

1.07 

0.10 

0.70 

EOC Algebra score (3 yravg) 

538.71 

11.65 

529.15 

16.51 

9.56 

** 

0.01 

537.85 

11.26 

0.86 

0.78 

EOC Biology score (3 yravg) 

542.04 

11.58 

536.46 

18.34 

5.58 


0.11 

541.11 

13.52 

0.93 

0.78 

EOC English score (3 yravg) 

533.07 

9.70 

532.03 

13.24 

1.04 


0.68 

534.10 

7.33 

-1.04 

0.65 

EOC History score (3 yravg) 

522.82 

7.97 

518.42 

13.31 

4.41 


0.08 

520.89 

8.63 

1.93 

0.38 

Attendance & Graduation 

Attendance rate (% days present) 

0.93 

0.02 

0.93 

0.03 

0.00 


0.99 

0.94 

0.01 

-0.01 

0.18 

Graduation rate 

0.89 

0.06 

0.86 

0.12 

0.04 


0.09 

0.91 

0.05 

-0.02 

0.26 




Consortium 

Schools 

(N=29) 

All Other Tennessee Schools 
(N^259) 

Matched Comparison Schools 
(N=29) 

Mean 

Stri. 

Dev. 

Mean 

Std. 

Dev. 

Difler. 

P 

value 

Mean 

Std. 

Dev. 

Difler. 

P 

value 

School Resources 

Student/ teac her ratio 

16.47 

2.75 

16.43 

2.28 

0.47 

0.94 

16.83 

2.93 

-0.36 

0.63 

Log of total expenditures per pupil 

8.99 

0.09 

9.05 

0.12 

-0.03 ** 

0.01 

9.00 

0.10 

-0.01 

0.68 

Log of sc hool size (enrollment in 

6.73 

0.59 

6.74 

0.56 

0.04 

0.91 

6.65 

0.58 

0.07 

0.64 

grades9-12) 











Community Characteristics 

%population with college degree 

0.16 

0.06 

0.21 

0.12 

-0.05 * 

0.04 

0.16 

0.10 

0.00 

0.85 

%population below poverty line 

0.15 

0.04 

0.13 

0.05 

0.02 

0.03 

0.16 

0.03 

0.00 

0.82 

Log of population's median annual 

10.51 

0.12 

10.66 

0.22 

-0.15 ** 

<0.01 

10.50 

0.15 

0.01 

0.88 

income 











School locale=rural 

0.38 

0.49 

0.53 

0.50 

-0.15 * 

0.13 

0.62 

0.49 

-0.24 

0.07 

Distance to nearest public college 

9.42 

6.69 

10.49 

8.08 

-1.07 

0.49 

9.57 

8.24 

-0.15 

0.94 

(miles) 











ClEand Advanced Placement 

Number of CTE prog ram area sat 

6.14 

1.43 

5.98 

1.04 

0.15 

0.47 

6.07 

0.96 

0.07 

0.83 

school 











Percent of graduates with CTE 

0.40 

0.21 

0.40 

0.20 

0.00 

0.98 

0.40 

0.18 

0.00 

0.98 

concentrations 











Number of AP courses at school 

4.55 

4.99 

4.48 

5.22 

0.08 

0.94 

3.69 

4.94 

0.86 

0.51 

Ratio of AP enrollments to 

0.22 

0.25 

0.23 

0.37 

-0.01 

0.89 

0.18 

0.29 

0.04 

0.58 

grades 11-12 












*The difference is statistically significant at the 95 percent confidence level using a two-tailed test. 

**7he difference is statistically significant at the 99 percent confidence level using a two-tailed test. 

Note : The p-valuesare based on a t-test of the difference between the Consortium schoolsand groupsof comparison schools. 
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Table 8 (above) illustrates the results from tests of the statistical significance of 
differences in characteristics of the Consortium schools and the non-Consortium 
schools before and after matching. Before matching, the Comparison schools and the 
non-Comparison schools show statistically significant differences in observed school 
characteristics representing demographic characteristics of the student population, 
test scores of the student population, attendance and graduation, school resources, 
community characteristics, and CTE and AP course taking. After matching, there are 
no statistically significant differences between the Consortium schools and the 
subset of 29 matched comparison schools on any of the observed school 
characteristics. 

We also calculated the absolute standardized bias (ASB) as another diagnostic of the 
balance between the treatment and control groups on the covariates of interest. 
Stuart (2007) recommends that absolute standardized bias values greater than 0.50 
are “particularly problematic,” and ideally should be less than 0.25. After matching, 
no covariates have an absolute standardized bias greater than 0.50. The following 
covariates, however, have an absolute standardized bias between 0.25 and 0.50 after 
matching: rural school locale (ASB=0.49), percent of students with Individualized 
Education Programs (ASB=0.34), attendance rate (ASB=0.30), and graduation rate 
(ASB=0.27). All school-level covariates from the propensity score model are 
controlled for in the statistical models used to estimate the impact of the 
intervention, in order to remove any bias caused by these differences, to reduce 
unexplained variation in outcomes, and to improve the precision of the impact 
estimates. 

Statistical models used to estimate the 
impactof the intervention 

This study uses two-level hierarchical models to accommodate clustering of 
observations that we assume is present in the data (e.g., intraclass correlations) and 
to ensure that standard errors are measured correctly. The models also allow us to 
account for covariates that may be correlated with the intervention condition and 
with outcomes. 

Model specification forthe continuous outcome 
va riable 

Composite ACT scores is the only continuous outcome variable. We estimate a two- 
level hierarchical linear model (HLM) with students nested in schools. The 
intervention indicator appears in level 2, the school level. The model is estimated as 
follows: 
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Level 1 model (student level): 

Yjj — Po, + Plj(Y*ij) + SPm/Xmij ) + e ij j 

where 

Yjj is the composite score on the ACT (Y) for student i, in school j, 

poj is the school-level intercept for school j, 

pij is the average pretest slope for students at school j, 

Y*jj is the pretest score for student i at school j, 

p m j are M coefficients corresponding to student-level demographic covariates, 

Xmij represents demographic characteristics of student z at school j for gender 
(dichotomous variable where male=0, female=l), minority race (dichotomous variable 
for whether the student is categorized as a race other than White, 0=no, l=yes), 
Limited English Proficiency status (0=no, l=yes), Individualized Education Program 
status (0=no, l=yes), free or reduced-price lunch status (0=no, l=yes), and 

Sjj is the random effect representing the difference between student if s score and the 
predicted mean for school j. These residual effects are assumed to be normally 
distributed with a mean of 0 and a variance of a 2 . 


Level 2 model (school level): 

Poj = Too + Yoi(Ij) + Ey 0q (Wqj ) + p 0 j , 
where 

Poj is the school-level intercept for school j, 

Yoo is the conditional school-level mean, 

Yoi is the coefficient corresponding to the intervention effect, 

Ij represents the intervention status for school j (l=Consortium school, 0=matched 
comparison school), 

Yo q are coefficients corresponding to the school-level covariates, 

\\ are Q school-level covariates for school j, which consist of the propensity score 
and all of the covariates included in the school-level propensity score model (baseline 
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measures for percent of students who are minorities, percent of students who are 
male, percent of students who receive free or reduced-price lunch, percent of 
students who are Limited English Proficiency status, percent of students with an 
Individualized Education Program, three-year average ACT scores in English, 
mathematics, reading, and science; three-year average end-of-course scores in 
algebra, biology, English, and history; average school attendance rate; high school 
graduation rate; student/teacher ratio; log of total expenditures per pupil; log of 
school size; percent of the population with a college degree; percent of the 
population below the poverty line; log of the population’s median annual income; 
rural school locale indicator; distance to nearest public college in miles; number of 
CTE program areas offered by the high school; percent of high school graduates with 
CTE concentrations; number of AP courses offered at the school; and ratio of 
enrollment in AP courses to total enrollment in grades 11 and 12; and 

g oj is the deviation of school j’s mean from the grand mean, conditional on 
covariates. This effect is assumed to be normally distributed with a mean of 0 and a 
variance of x 2 . 


Combined model 

The combined model is: 

Yjj = Yoo + Yoi(Ij) + £yo q (Wqj ) + pij(Y*ij) + SPmjOCmij ) + |t 0 j + Sij, 

where the coefficient of interest is y 0 i, which represents the difference between the 
Consortium schools’ and matched comparison schools’ conditional means. The 
intervention variable (I) is a dichotomous variable that indicates whether the student 
was enrolled in a Consortium school (1=1) or a matched comparison school (1=0). We 
examine whether the intervention effect is statistically significant using a two-tailed 
test and an alpha level of 0.05. Effect sizes are calculated using Cohen’s d with the 
pooled standard deviation. 

Model specification forthe die ho to mo us outcome 
va riables 

The dependent variables for the remaining research questions are all dichotomous 
outcomes: enrolling in an AP course, earning a score of 3 or higher on an AP exam, 
enrolling in college, and persisting in college. We use the same covariates described 
previously, but replace the dependent variable with the binary outcome and estimate 
the model using a logit link function. The hierarchical generalized linear model is 
estimated as: 


hij = Too + Yoi(Ij) + Xyoq (Wqj ) + Plj(Y*ij) + XpmjOCmij ) + P Oj , 
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r|ij is the log of the odds of success on the outcome of interest for student z in school 
j, 


Yoo is the school-level intercept, 

Ij represents the treatment condition (intervention [1] versus comparison [0]), 

W is the vector for school-level covariates, 

X is the vector for student-level demographic covariates, 
and ji 0 j is the unmodeled residual for school j. 

The coefficient on the intervention variable in the model measures the marginal 
impact of being in the intervention group on the log of the probability of each 
outcome (enrolling in an AP course, earning a score of 3 or higher on an AP exam, 
enrolling in college, and persisting in college). Impacts are estimated using the 
regression equation to calculate each student’s probability of successful completion 
of each outcome using individual student characteristics, and then generating group- 
level means for the intervention and comparison groups using the intervention 
variable. The impact estimates are calculated by the first difference between the 
intervention and comparison groups. 

Missing data 

We use case-wide deletion of observations with missing outcome measures or 
baseline equivalence pretest data. This method is appropriate when the outcome data 
are missing for students within schools in studies where the intervention is 
administered at the school level (Puma, Olsen, Bell, & Price, 2009). All of the pretest 
and outcome data for the impact analyses are available from state administrative 
records so there is no missing data at the school level. Some data at the student level 
are missing due to incomplete administrative records and student attrition (see table 
9, below). 

The data for the college enrollment outcome includes records only for students who 
attended a postsecondary institution, and it is difficult to determine which of the 
remaining students did not attend college at all and which ones were lost from the 
sample due to attrition. Attrition occurs when students leave Tennessee’s public 
school system. This can occur due to withdrawal to a state institution (e.g., 
Tennessee Department of Human Services), transfer to out-of-state schools, transfer 
to non-public schools in Tennessee, doctor-certified health withdrawal, death, 
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transfer to mental or drug rehabilitation institute, transfer to a home school, or 
issuance of a court order. Attrition may also be for unknown reasons, if students 
have no withdrawal code or are missing from enrollment records after the baseline 
year. We categorize students as “missing” from the analysis sample of the college 
enrollment outcome if they do not have school enrollment or transcript records in 
grade 12, or if they have data indicating that they withdrew from a Tennessee public 
high school prior to the end of grade 12 for reasons other than dropout. All other 
students who do not have enrollment records at a postsecondary institution by the 
fall after high school graduation are categorized as not enrolling in college. This 
includes students who do not graduate from high school due to dropout. 

There are low levels of missing data for the covariates in the statistical models 
because these data are also available from state administrative records. Some of 
these variables, such as date of birth and gender, are also self-reported on state 
assessments. The self-reported data are used for students missing records in the 
demographic files provided by the Tennessee Department of Education. The dummy 
variable adjustment method is used for remaining missing values for covariates. 
Using this approach, the value of the missing independent variables is set to a 
constant value of 0, and an additional dummy variable is added to the model to 
indicate whether the actual value is missing. 
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Table 9. Summary of attrition forthe Consortium and comparison groups, by outcome. 






Consortium group 



Comparison group 


Outcome 

ST 

Cohort 

Full 

Sample 

N 

Analysis 
Sample N 

Difference 

Attrition 

Rate 

Full 

Sample 

N 

Analysis 
Sample N 

Difference 

Attrition 

Rate 

Composite 
ACTscoies 
in grade 11 

2013/ 

14 

Grade 

9 

6,430 

4,139 

2,291 

35.6% 

6,336 

4,300 

2,036 

32.1% 

Composite 

ACTscoies 

2012/ 

13 

Grade 

11 

6,350 

4,510 

1,840 

29.0% 

5,814 

4,477 

1,337 

23.0% 

in grade 12 

2013/ 

14 

Grade 

10 

6,430 

4,644 

1,786 

27.8% 

6,206 

4,765 

1,441 

23.2% 


2014/ 

15 

Grade 

9 

6,430 

4,356 

2,074 

32.3% 

6,336 

4,425 

1,911 

30.2% 

Enrollment 
in one or 

2012/ 

13 

Grade 

11 

6,350 

4,905 

1,445 

22.8% 

5,814 

4,625 

1,189 

20.5% 

mote AP 

courses 

2013/ 

14 

Grade 

10 

6,430 

5,134 

1,296 

20.2% 

6,206 

4,968 

1,238 

19.9% 


2014/ 

15 

Grade 

9 

6,430 

5,076 

1,354 

21.1% 

6,336 

5,002 

1,334 

21.1% 

Score of 3 
or higher on 

2012/ 

13 

Grade 

11 

6,350 

4,905 

1,445 

22.8% 

5,814 

4,625 

1,189 

20.5% 

an AP 

exam 

2013/ 

14 

Grade 

10 

6,430 

5,134 

1,296 

20.2% 

6,206 

4,968 

1,238 

19.9% 


2014/ 

15 

Grade 

9 

6,430 

5,076 

1,354 

21.1% 

6,336 

5,002 

1,334 

21.1% 
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Consortium group 



Comparison group 


Outcome 

SY 

Cohort 

Full 

Sample 

N 

Analysis 
Sample N 

Difference 

Attrition 

Rate 

Full 

Sample 

N 

Analysis 
Sample N 

Difference 

Attrition Rate 

College 

enrollment 

Fall 

2012 

Grade 

12 

6,051 

4,534 

1,517 

25.1% 

5,583 

4,723 

860 

15.4% 


Fall 

2013 

Grade 

11 

6,350 

4,905 

1,445 

22.8% 

5,814 

4,625 

1,189 

20.5% 


Fall 

2014 

Grade 

10 

6,430 

5,134 

1,296 

20.2% 

6,207 

4,968 

1,239 

20.0% 


Fall 

2015 

Grade 

9 

6,430 

5,539 

891 

13,9% 

6,336 

5,454 

882 

13.9% 

College 

persistence 

Fall 

2014 

Grade 

11 

6,039 

4,623 

1,416 

23.4% 

5,500 

4,339 

1,161 

21.1% 


Fall 

2015 

Grade 

10 

6,430 

4,957 

1,473 

22.9% 

6,207 

4,814 

1,393 

22.4% 
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Appendix C: Supplemental tables 


This appendix provides supplemental tables for the report. Table 10 provides full 
results from the estimates of the impact of the intervention, as presented for the 
NEi3 report. 

Table 11 shows a crosswalk of the indicators comprising the components for the 
fidelity of implementation for each year of the grant. 
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Table 10. Full resultsfrom the estimates of the impact of the intervention. 
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CONFIRMATORY CONTRASTS 


Grade 10 College ACT 

cohort readiness composite 

score - 
grade 12 

29 

4,644 

29 

4,765 

5.029 

4.968 

A 

19.641 

0.246 

0.049 

0.087 

0.004 

9,407 

AP Score 3or 

performance higheron 

AP exam 

29 

5,134 

29 

4,968 

0.304 

0.284 

A 

0.089 

0.938 

NR 

0.377 

0.013 

10,101 

College Enrollment 

enrollment in college 

by fall after 
grade 12 

29 

5,134 

29 

4,968 

0.494 

0.498 

A 

0.543 

0.138 

NR 

0.059 

0.020 

10,101 

Grade 11 College Enrollment 

cohort persistence in college 2 

consecut- 
ive fa II 
semesters 
afterfall of 
grade 12 

29 

4,623 

29 

4,339 

0.494 

0.489 

A 

0.395 

0.130 

NR 

0.065 

0.047 

8,962 
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EXPLORATORY CONTRASTS 














Grade 9 
cohort 

College 

readiness 

ACT 

composite 
score - 
grade 11 

29 

4,139 

29 

4,300 

4.779 

4.732 

A 

19.023 

-0.086 

-0.018 

0.125 

0.492 

8,437 

Grade 10 
cohort 

AP 

participation 

Enrollment 
in AP 

course 

29 

5,134 

29 

4,968 

0.448 

0.411 

A 

21.479 

1.204 

nr 

0.341 

<0.001 

10,100 

Grade 10 
cohort; 
enrolled in 
APcourse 

AP 

performance 

Score 3 or 
hig heron 
AP exam 

29 

1,433 

29 

1,073 

0.484 

0.492 

B 

0.408 

0.670 

nr 

0.308 

0.030 

2,508 

Grade 11 
cohort 

College 

readiness 

ACT 

composite 
score - 
grade 12 

29 

4,477 

29 

4,510 

5.046 

4.975 

A 

19.467 

0.019 

0.004 

0.063 

0.76 

8,985 

Grade 11 
cohoit 

AP 

performance 

Score 3 or 
hig heron 
AP exam 

29 

4,905 

29 

4,625 

0.293 

0.282 

A 

0.087 

0.098 

NR 

0.28 

0.726 

9,528 


















Sample 

Contrast Name 

Post test Measure Name 

Treatment G roup 

Treatment Group N of 

Students 

Comparison Group N of 

Clusters 

Comparison Group N of 

Students 

Treatment Group SD 

Comparison Group SD 

Standard Deviation Source 

(Code) 3 

Comparison Group Mean 

Impact Estimate 

Standardized Effect Size 

(optional) 

Impact Standard Error 

p- value 

Degrees of Freedom 

Grade 11 
cohort; 
enrolled in 
APcourse 

AP 

performance 

Score 3 or 
hig heron 
AP exam 

29 

1,262 

29 

1,059 

0.483 

0.482 

B 

0.367 

-0.299 

NR 

0.518 

0.564 

2,319 

Grade 11 
cohoit 

AP 

participation 

Enrollment 
in AP 

course 

29 

4,905 

29 

4,625 

0.437 

0.42 

A 

0.229 

0.826 

NR 

0.387 

0.033 

9,528 

Grade 11 
cohort 

College 

enrollment 

Enrollment 
in college 
by fall after 
grade 12 

29 

4,905 

29 

4,625 

0.493 

0.496 

A 

0.563 

0.005 

NR 

0.061 

0.928 

9,528 

Grade 12 
cohoit 

College 

enrollment 

Enrollment 
in college 
by fall after 
grade 12 

29 

4,534 

29 

4,723 

0.489 

0.497 

A 

55.537 

0.261 

NR 

0.119 

0.028 

9,257 


















VJ 



EXTENSION 


Grade 

9 

cohort 

College 

readiness 

ACT 

composite 
score - 
grade 12 

29 

4,356 

29 

4,425 

5.031 

4.950 

A 

0.074 

-0.034 

NR 

0.119 

0.772 

8,781 


AP 

Participation 

Enrollment 
in AP 

course 

29 

5,076 

29 

5,002 

0.441 

0.414 

A 

0.219 

0.817 

NR 

0.197 

<0.001 

10,076 


AP 

performance 

Score 3 or 
hig heron 

AP exam 

29 

5,076 

29 

5,002 

0.315 

0.283 

A 

0.088 

0.857 

NR 

0.356 

0.016 

10,076 


College 

enrollment 

Enrollment 
in college 
by fall after 
grade 12 

29 

5,539 

29 

5,454 

0.484 

0.491 

A 

0.592 

0.089 

NR 

0.057 

0.120 

10,811 

Grade 

10 

cohort 

College 

persistence 

Enrollment 
in college 2 
consecut- 
ive fa II 
semesters 

29 

4,957 

29 

4,814 

0.491 

0.485 

A 

0.379 

0.133 

NR 

0.058 

0.022 

9,770 


a Source code for the standard deviations: A=student-level standard deviations calculated from the full sample; B=student-level 
standard deviationscalculated from study data, specific to the subgroup analyzed in the current contrast. 

NR=not reported 


















Table 11. Crosswalk of fidelity indicatorsforeach program component in SYs2011/12 and 2012/13 versus SY 2013/14. 



Indicator by School Year (SY) 

Component 

SYs 2011/ 12 and 2012/ 13 

Action 
la ken 

SY 2013/ 14 

1) Management and 

1. Directorconvenesthe Advisory Board 

None 

1. Directorconvenesthe Advisory Board 

Communication 

2. Attendance is high at Board meetings 

None 

2. Attendance is high at Board meetings 


3. Directorcommunicates with Consortium 
staff 

None 

3. Leadership Team meets and shares 
information with staff 


4. Directorcommunicates with school systems 

None 

4. Leadership Team communicates with 
school systems 


5. Directorcommunicates with the evaluator 

Removed 

— 


6. Directorprovidesoutreach to othergroups 
and legislative bodies 

None 

5. Leadership/Counselors Tea ms pro vide 
outreach to othergroupsand legislative 
bodies 


7. Staff participate in weekly meetings 

Removed 

— 


8. Staff receive professional development 

None 

6. Staff receive professional development 


— 

Moved from 
7.2 

7. Schools collaborate with one another 


— 

New 

8. Conduct course review 

2) College and Career 
Counselors 

1. Assign Lead Counselor and staff to high 
schools 

Removed 

— 


2. Spend at least 1 day perweekin high 
school 

None 

1. Spend at least 1 day per week at each 
school 


3. Conduct core activities in high school 

None 

2. Assist students (individually or in groups) 
with developing educational, career, and life 
planning 


4. Provide individual college/career 
counseling to students 

Collapsed 
into 2.2 

— 
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Indicator by School Year (SY) 

Component 

SYs 2011/ 12 and 2012/ 13 

Action 
la ken 

SY 2013/ 14 


5. Make group or class presentation to 
students 

Removed 

— 


6. Create and distribute informational 
materials 

Removed 

— 


7. Organize college visits 

Reworded 

3. Conduct onsite and/orvirtual college visits 
during the school day 


8. Bring family members on college visits 

Reworded 

4. Conduct virtual and/oronsite college visits 
outside of school calendar 



New 

5. Collaborate with educators, community 
members, and parents to assist students with 
educational, career, and life planning 



New 

6. Collaborate with other counselors and 
educators in the school-wide integration of 
the guidance services 

3) Learning Resources 
learn: Quality of 

1. Maintain instructional spec ialists-to-school 
district ratio 

Removed 

— 

Insbuction 

2. Organize events to share best practices 

Moved 

[Similarto 1.5] 


3. Provide ongoing support to teachersof AP, 
online, and distance learning 

Moved 

[Similar to 4.6, 5.2] 


4. Evaluate online coursesforpossible 
adoption 

Moved 

[Similarto 4.4] 


5. Create Consortium-wide course directory 
and monitorcourse availability 

Moved 

[Similarto 6.4] 


— 

New 

1. Coordinate collaborative professional 
development events a nd/or re sources 


— 

New 

2. Coordinate online collaborative 
professional communities 


— 

New 

3. Coordinate training foronline teachers 




Indicator by School Year (SY) 

Component 

SYs 2011/ 12 and 2012/ 13 

Action 
la ken 

SY 2013/ 14 


— 

New 

4. Assist schools with STEM curriculum and 
instruction 

4) Learning Resources 

1. Expand online enrollment 

None 

1. Expand online enrollments 

leam: Distance and 
Online "technology 

2. Expand distance learning enrollment 

None 

2. Expand distance learning enrollments 

3. Monitortechnology in schools and assist in 
using technology 

Moved 

[Similar to 6.1] 


4. Identify and acquire technology to support 
distance and online learning 

Moved 

[Similarto 6.1] 


5. Ensure reliability of equipment 

Moved 

[Similarto 6.1] 


6. Create distance and online handbook 

Moved 

[Similarto 6.3] 


7. Maintain Consortium-wide online learning 
center 

Moved 

[Similarto 6.3] 


— 

Moved from 
3.1 

3. Recruit/ retain and supervise online 
teachers 


— 

New 

4. Coordinate online course development 


— 

New 

5. Coordinate online course evaluation 


— 

Moved from 
3.3 

6. Assist/train school personnel (including 
teachers) in using technology 


— 

New 

7. Ensure schools use ACTPrep online 
software 

5) Learning Resources 
leam: 

1. Expand enrollments in dual enrollment 
offerings 

None 

1. Expand dual enrollment enrollments 

College- Level Courses 

— 

Moved from 
3.3 

2. Assist in the implementation of AP programs 


2. Support supply and demand review 

Removed 

— 


3. Provide professional development to 
Consortium staff 

Removed 

— 
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Indicator by School Year (SY) 

Component 

SYs 2011/ 12 and 2012/ 13 

Action 
la ken 

SY 2013/ 14 


— 

New 

3. Promote ormarket AP courses 


— 

Moved from 
7.3 

4. Expand AP enrollments 


— 

New 

5. Coordinate dual enrollment with 
postsecondary institutions 

6) Resources and 
Services 

1. Purchase and maintain technology for 
distance and online learning 

None 

1. Purchase, maintain, and monitor distance 
leaming/conferencing software in schools 


2. Provide Internet access to support distance 
and online learning 

Removed 

— 


— 

New 

2. Maintain openSIS (student information 
system) for online course tracking 


3. Maintain a functioning distance learning 
server 

Removed 

— 


— 

New 

3. Maintain learning center website 


4. Provide teacherand staff trainings 

Moved 

[Similar to 4.6] 


— 

Moved from 
3.5 

4. Create/update course directory as needed 

7) LEA Ra rlners 

1. Schoolscollaborate with Consortium staff 

Moved 

[Similarto 3.4, 4.6, and 5.2] 


2. Schoolscollaborate with otherConsortium 
schools 

Moved 

[Similarto 1.7] 


3. Schools expand AP enrollments 

Moved 

[Similarto 5.4] 


4. Schools provide access to Consortium 
courses 

Removed 

— 


5. Schools provide teachers or staff access to 
Consortium training 

Moved 

[Similar 3.1] 


6. LEAspickup recurring costs 

Removed 

— 
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